请选择 进入手机版 | 继续访问电脑版
楼主: neuroexplorer
824 0

[学习分享] Running R jobs quickly on many machines [推广有奖]

  • 5关注
  • 23粉丝

学科带头人

79%

还不是VIP/贵宾

-

威望
0
论坛币
29106 个
通用积分
844.6045
学术水平
53 点
热心指数
70 点
信用等级
58 点
经验
176572 点
帖子
3222
精华
0
在线时间
1394 小时
注册时间
2013-7-21
最后登录
2024-3-17

+2 论坛币
k人 参与回答

经管之家送您一份

应届毕业生专属福利!

求职就业群
赵安豆老师微信:zhaoandou666

经管之家联合CDA

送您一个全额奖学金名额~ !

感谢您参与论坛问题回答

经管之家送您两个论坛币!

+2 论坛币
Running R jobs quickly on many machinesPosted onJanuary 22, 2016AuthorJohn MountCategoriesExciting Techniques, Practical Data Science, Pragmatic Data Science, Pragmatic Machine Learning,ProgrammingTagsEC2, parallel programming, R

As we demonstrated in “A gentle introduction to parallel computing in R” one of the great things about R is how easy it is to take advantage of parallel processing capabilities to speed up calculation. In this note we will show how to move from running jobs multiple CPUs/cores to running jobs multiple machines (for even larger scaling and greater speedup). Using the technique on Amazon EC2 even turns your credit card into a supercomputer.




Colossus supercomputer : The Forbin Project


R itself is not a language designed for parallel computing. It doesn’t have a lot of great user exposed parallel constructs. What saves us is the data science tasks we tend to use R for are themselves are very well suited for parallel programming and many people have prepared very goodpragmatic libraries to exploit this. There are three main ways for a user to benefit from library supplied parallelism:

  • Link against superior and parallel libraries such as the Intel BLAS library (supplied on Linux, OSX, and Windows as part of theMicrosoft R Open distribution of R). This replaces libraries you are already using with parallel ones, and you get a speed up for free (on appropriate tasks, such as linear algebra portions of lm()/glm()).
  • Ship your modeling tasks out of R into an external parallel system for processing. This is strategy of systems such as rx methods from RevoScaleR, now Microsoft Open R, h2o methods from h2o.ai, orRHadoop.
  • Use R’s parallel facility to ship jobs to cooperating R instances. This is the strategy used in “A gentle introduction to parallel computing in R”and many libraries that sit on top of parallel. This is essentially implementing remote procedure call through sockets or networking.

We are going to write more about the third technique.

The third technique is essentially very course grained remote procedure call. It depends on shipping copies of code and data to remote processes and then returning results. It is ill suited for very small tasks. But very well suited a reasonable number of moderate to large tasks. This is the strategy used by R’s parallel library and Python‘s multiprocessinglibrary (though with Python multiprocessing you pretty much need to bring in additional libraries to move from single machine to cluster computing).

This method may seem less efficient and less sophisticated than shared memory methods, but relying on object transmission means it is in principle very easy to extend the technique from a single machine to many machines (also called “cluster computing”). This is what we will demonstrate the R portion of here (in moving from a single machine to a cluster we necessarily bring in a lot of systems/networking/security issues which we will have to defer on).

Here is the complete R portion of the lesson. This assumes you already understand how to configure “ssh” or have a systems person who can help you with the ssh system steps.

Take the examples from “A gentle introduction to parallel computing in R” and instead of starting your parallel cluster with the command: “parallelCluster <- parallel::makeCluster(parallel::detectCores()).”

Do the following:

Collect a list of addresses of machines you can ssh. This is the hard part, depends on your operating system, and something you should get help with if you have not tried it before. In this case I am using ipV4 addresses, but when using Amazon EC2 I use hostnames.

In my case my list is:

  • My machine (primary): “192.168.1.235”, user “johnmount”
  • Another Win-Vector LLC machine: “192.168.1.70”, user “johnmount”

Notice we are not collecting passwords, as we are assuming we have set up proper “authorized_keys” and keypairs in the “.ssh” configurations of all of these machines. We are calling the machine we are using to issue the overall computation “primary.”

It is vital you try all of these addresses with “ssh” in a terminal shell before trying them with R. Also the machine address you choose as “primary” must be an address the worker machines can use reach back to the primary machine (so you can’t use “localhost”, or use an unreachable machine as primary). Try ssh by hand back and forth from primary to all of these machines and from all of these machines back to your primary before trying to use ssh with R.

Now with the system stuff behind us the R part is as follows. Start your cluster with:

primary <- '192.168.1.235'machineAddresses <- list(  list(host=primary,user='johnmount',       ncore=4),  list(host='192.168.1.70',user='johnmount',       ncore=4))spec <- lapply(machineAddresses,               function(machine) {                 rep(list(list(host=machine$host,                               user=machine$user)),                     machine$ncore)               })spec <- unlist(spec,recursive=FALSE)parallelCluster <- parallel::makeCluster(type='PSOCK',                                         master=primary,                                         spec=spec)print(parallelCluster)## socket cluster with 8 nodes on hosts##                   ‘192.168.1.235’, ‘192.168.1.70’

And that is it. You can now run your job on many cores on many machines. For the right tasks this represents a substantial speedup. As always separate your concerns when starting: first get a trivial “hello world” task to work on your cluster, then get a smaller version of your computation to work on a local machine, and only after these throw your real work at the cluster.

As we have mentioned before, with some more system work you canspin up transient Amazon ec2 instances to join your computation. At this point your credit card becomes a supercomputer (though you do have to remember to shut them down to prevent extra expenses!).



二维码

扫码加我 拉你入群

请注明:姓名-公司-职位

以便审核进群资格,未注明则拒绝

关键词:Machines machine running Quickly Quick R programming Parallel computing

您需要登录后才可以回帖 登录 | 我要注册

本版微信群
加好友,备注cda
拉您进交流群

京ICP备16021002-2号 京B2-20170662号 京公网安备 11010802022788号 论坛法律顾问:王进律师 知识产权保护声明   免责及隐私声明

GMT+8, 2024-3-29 09:16