Ruby Programming/Running Multiple Processes

= Running Multiple Processes =

There are several ways to run external commands in Ruby.

output = `command here` # gives you back full stdout

stdout_and_stdin = IO.popen("command here")

require 'popen3' # require 'open3' in 1.9

input,output,error,running_thread_on_19_or_greater = Open3.popen3("command here")

Open3.popen3("command here") do |stdin, stdout, stderr| # ... end
 * 1) or the same:

pid = fork { puts 'in child process' } # posix platforms only

pid = Process.spawn "ls" # 1.9.x only pid = Process.daemon "ls" # 1.9.x only, basically does a spawn and a disassociate on that pid.

(1.9 only) has many options.

Accessing PID's of running processes
For many of these commands there is no way to get the pid of a running child process while it is still running. For  you get it immediately via   as well as the fork return value, as well as for   where the PID is the return value.

If you want the pid and I/O of a running sub-process, you'll have to use fork and redirect the child processes I/O to previously created pipe's, or jruby users have an available

pid,input,output,error = IO.popen4("ls") #jruby only method or use the  method if it is available, ex: input, output, error, thread_if_on_19 = Open3.popen3 "ls" pid = thread.pid io = IO.popen("ls") pid = io.pid

The PID is not accessible for the  and   calls while they are running (since $? is only available on a per-thread basis, and they haven't finished).

In reality all that $? means is "tell me what my child process was that most recently finished."

Note that on 1.8  doesn't have the PID available. If you want the pid in 1.8 in Linux, you'll probably need to create some pipes, fork a new process, and redirects its pipes to the ones you created. For windows users, though, if you want the PID for an open3 call in 1.8 windows you'll want to use a helper gem or IO.popen4 on jruby.

If you wanted to start a sub process without spawning a new thread in 1.9 (like popen3 but without its extra thread) you could use Process.spawn or create pipes, fork a process, and redirects its IO to the pipes et al.

If you want to start a sub process without spawning a new thread or creating IO objects in windows 1.8, you'll need to use a helper gem, like the win32-process gem.

Note well that if you start a process with access to its stdout/stdin/stderr streams, if you do not read from those streams in some way the process can *block* after it fills up a stream's buffer. I.e. you must read from them if they give a lot of output.

Reading piece-wise from a stream
This code seems to work: while !out.eof? print out.read 1024 end Process.wait out.pid } Though you probably don't have to wait for the PID at the end.

You could also just read the full stream output thus: print out.read

Reading and writing to an external process
You can use popen3 if you want to read and write to an external process. You can do it with popen as well by using the open stream type of "r+" (not "rw").

Note also that with popen that by default on windows it opens up all your file streams in "ascii" mode. To set them as binary mode, if desired, call #binmode on each descriptor that comes back to you (thanks imagemagick guys for the example).

Ex:

IO.popen("ruby", "r+") do |pipe| pipe.puts "puts 10**6" pipe.puts "__END__" pipe.gets end

Only writing to a command/using encoding
Most of the other examples here expose both a read and write pipe to the child process. Sometimes you want to have just one or the other. HEre is a way.

just_stdin_to_process = open("|process_name", "w")

You can also use this same method to set an encoding for strings coming in from a process (there may be other ways, as well), in 1.9.

just_stdin_to_process = open("|process_name", "r:UTF-8")

It might work to open like this, too

stdin_and_out = IO.popen(c, "w") # outputs to stdout, stderr, but you can write to it

Windows: how to run ruby without opening up a command window
In general, if you start an app using rubyw.exe instead of ruby.exe, your ruby app's stdout/stderr is piped to (windows' equivalent of) /dev/null, so no command window appears.

However, if, within that app, you make a call out to system("something_else.exe"), it will then pop up its own console for input/output, in case it needs any (unless it also has an equivalent of rubyw.exe available for you).

See http://www.ruby-forum.com/topic/213521 for a list of possible solutions. NB that the require 'win32/system' snippet is to use the win32-system gem, which you'd need to install first. There doesn't seem to be a way to do it with the stdlib currently.

Another option is to use ffi to call CreateProcess directly, to essentially the same effect: https://gist.github.com/rdp/8229520 (originally from https://gist.github.com/jarib/280865 so the childprocess gem might do some of this for you or you could roll your own).

Another option to rubyw.exe is to run it using ruby.exe but in a "minimized" window to lessen output: http://www.justskins.com/forums/winapp-without-console-window-97080.html

win32-open3 gem appears to be another option: http://stackoverflow.com/questions/12684463/ruby-system-call-on-windows-without-a-popup-command-prompt

jrubyw also seems to be able to run them without opening a new command prompt window (jrubyw.exe).

Also in windows you may be able to spawn a separate background process using "start.exe" like system("start my_command.exe") ref: http://stackoverflow.com/a/3840737/32453

Chaining processes
So say you want to chain some processes together in ruby, the equivalent of bashes `a | b | c`?

Well first, you could run just that: `a | b | c` and it will hand the string over to bash and let it do all the redirection and return you the output of stdout. You can even run popen with chained commands, and it will do the same.

Or you can roll your own. The basic way is that you want to open some pipe's, then fork, within the child redirect stdin/stdout to the appropriate pipe (in our example case, that would be two pipes per connection, so total 6), then exec the desired command within each child process. phew!

Ruby has a built-in "shell" class. See also section "method 8". shows the basic syntax it uses internally for its redirection, I believe.

The gist of it is something like

1.9's Process.spawn with its :stdout and :stderr simplified syntax might make this even easier, and hopefully possible even in windows. For 1.8 windows you could probably use the win32/process gem which also provides simplified :stdout and :stderr redirection.

See also 1.9's Open3.pipeline_start