2012-05-20 14:20:11,799 PID: 1129 config.py:551 - DEBUG - Loading config 2012-05-20 14:20:11,800 PID: 1129 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 14:20:11,802 PID: 1129 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1, 'port': None} 2012-05-20 14:20:15,571 PID: 1129 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 14:20:15,571 PID: 1129 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 14:20:17,731 PID: 1129 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 14:20:17,732 PID: 1129 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 14:20:18,346 PID: 1129 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 14:20:18,347 PID: 1129 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 14:20:20,119 PID: 1129 cluster.py:1140 - INFO - Waiting for cluster to terminate... 2012-05-20 14:20:21,303 PID: 1129 cluster.py:1377 - INFO - Removing @sc-sfmpc placement group 2012-05-20 14:20:22,035 PID: 1129 cluster.py:1380 - INFO - Removing @sc-sfmpc security group 2012-05-20 14:20:25,588 PID: 1131 config.py:551 - DEBUG - Loading config 2012-05-20 14:20:25,589 PID: 1131 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 14:20:25,591 PID: 1131 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1, 'port': None} 2012-05-20 14:20:27,260 PID: 1131 start.py:176 - INFO - Using default cluster template: sfmcluster 2012-05-20 14:20:27,261 PID: 1131 cluster.py:1539 - INFO - Validating cluster template settings... 2012-05-20 14:20:32,171 PID: 1131 cluster.py:1555 - INFO - Cluster template settings are valid 2012-05-20 14:20:32,171 PID: 1131 cluster.py:1427 - INFO - Starting cluster... 2012-05-20 14:20:32,171 PID: 1131 cluster.py:952 - INFO - Launching a 1-node cluster... 2012-05-20 14:20:32,172 PID: 1131 cluster.py:1004 - INFO - Launching master node (ami: ami-4583572c, type: cg1.4xlarge)... 2012-05-20 14:20:32,765 PID: 1131 awsutils.py:165 - INFO - Creating security group @sc-sfmpc... 2012-05-20 14:20:38,754 PID: 1131 awsutils.py:275 - INFO - Creating placement group @sc-sfmpc... 2012-05-20 14:20:40,834 PID: 1131 cluster.py:772 - INFO - SpotInstanceRequest:sir-f4f06611 2012-05-20 14:20:40,834 PID: 1131 utils.py:93 - INFO - Starting cluster took 0.144 mins 2012-05-20 14:20:45,376 PID: 1133 config.py:551 - DEBUG - Loading config 2012-05-20 14:20:45,377 PID: 1133 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 14:20:45,378 PID: 1133 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1, 'port': None} 2012-05-20 14:20:48,145 PID: 1133 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 14:20:48,145 PID: 1133 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 14:20:52,114 PID: 1134 config.py:551 - DEBUG - Loading config 2012-05-20 14:20:52,114 PID: 1134 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 14:20:52,116 PID: 1134 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1, 'port': None} 2012-05-20 14:20:54,585 PID: 1134 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 14:20:54,585 PID: 1134 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 14:21:03,351 PID: 1136 config.py:551 - DEBUG - Loading config 2012-05-20 14:21:03,351 PID: 1136 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 14:21:03,353 PID: 1136 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1, 'port': None} 2012-05-20 14:21:06,274 PID: 1136 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 14:21:06,275 PID: 1136 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 14:22:12,272 PID: 1143 config.py:551 - DEBUG - Loading config 2012-05-20 14:22:12,272 PID: 1143 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 14:22:12,274 PID: 1143 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1, 'port': None} 2012-05-20 14:22:16,363 PID: 1143 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 14:22:16,363 PID: 1143 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 14:22:35,004 PID: 1147 config.py:551 - DEBUG - Loading config 2012-05-20 14:22:35,004 PID: 1147 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 14:22:35,011 PID: 1147 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1, 'port': None} 2012-05-20 14:22:38,278 PID: 1147 awsutils.py:1141 - INFO - Current price: $0.67 2012-05-20 14:22:38,279 PID: 1147 awsutils.py:1142 - INFO - Max price: $2.65 2012-05-20 14:22:38,279 PID: 1147 awsutils.py:1143 - INFO - Average price: $0.75 2012-05-20 14:22:38,279 PID: 1147 awsutils.py:1159 - INFO - 2012-05-20 14:22:38,279 PID: 1147 awsutils.py:1160 - INFO - Starting StarCluster Webserver... 2012-05-20 14:22:38,280 PID: 1147 awsutils.py:1166 - INFO - Server address is http://127.0.0.1:54196 2012-05-20 14:22:38,280 PID: 1147 awsutils.py:1168 - INFO - (use CTRL-C or navigate to http://127.0.0.1:54196/shutdown to shutdown server) 2012-05-20 14:22:38,280 PID: 1147 webtools.py:190 - INFO - Browsing http://127.0.0.1:54196/spothistory.html using 'default'... 2012-05-20 14:22:38,821 PID: 1147 webtools.py:56 - INFO - Shutting down server... 2012-05-20 14:22:49,653 PID: 1151 config.py:551 - DEBUG - Loading config 2012-05-20 14:22:49,653 PID: 1151 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 14:22:49,655 PID: 1151 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1, 'port': None} 2012-05-20 14:22:53,212 PID: 1151 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 14:22:53,212 PID: 1151 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 14:25:43,797 PID: 1164 config.py:551 - DEBUG - Loading config 2012-05-20 14:25:43,798 PID: 1164 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 14:25:43,799 PID: 1164 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1, 'port': None} 2012-05-20 14:25:47,283 PID: 1164 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 14:25:47,283 PID: 1164 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 14:32:36,510 PID: 1195 config.py:551 - DEBUG - Loading config 2012-05-20 14:32:36,511 PID: 1195 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 14:32:36,512 PID: 1195 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1, 'port': None} 2012-05-20 14:32:40,574 PID: 1195 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 14:32:40,575 PID: 1195 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 15:15:33,823 PID: 1385 config.py:551 - DEBUG - Loading config 2012-05-20 15:15:33,824 PID: 1385 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 15:15:33,825 PID: 1385 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1, 'port': None} 2012-05-20 15:15:37,624 PID: 1385 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 15:15:37,625 PID: 1385 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 15:15:43,145 PID: 1389 config.py:551 - DEBUG - Loading config 2012-05-20 15:15:43,146 PID: 1389 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 15:15:43,147 PID: 1389 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1, 'port': None} 2012-05-20 15:15:45,511 PID: 1389 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 15:15:45,512 PID: 1389 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 15:15:46,910 PID: 1389 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 15:15:46,911 PID: 1389 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 15:15:47,442 PID: 1389 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 15:15:47,442 PID: 1389 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 15:15:47,997 PID: 1389 cluster.py:1368 - INFO - Canceling spot instance request: sir-f4f06611 2012-05-20 15:15:49,620 PID: 1389 cluster.py:1140 - INFO - Waiting for cluster to terminate... 2012-05-20 15:15:50,655 PID: 1389 cluster.py:1377 - INFO - Removing @sc-sfmpc placement group 2012-05-20 15:15:51,206 PID: 1389 cluster.py:1380 - INFO - Removing @sc-sfmpc security group 2012-05-20 15:16:07,196 PID: 1394 config.py:551 - DEBUG - Loading config 2012-05-20 15:16:07,196 PID: 1394 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 15:16:07,198 PID: 1394 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1, 'port': None} 2012-05-20 15:16:08,717 PID: 1394 start.py:176 - INFO - Using default cluster template: sfmcluster 2012-05-20 15:16:08,718 PID: 1394 cluster.py:1539 - INFO - Validating cluster template settings... 2012-05-20 15:16:13,062 PID: 1394 cluster.py:1555 - INFO - Cluster template settings are valid 2012-05-20 15:16:13,062 PID: 1394 cluster.py:1427 - INFO - Starting cluster... 2012-05-20 15:16:13,063 PID: 1394 cluster.py:952 - INFO - Launching a 1-node cluster... 2012-05-20 15:16:13,063 PID: 1394 cluster.py:979 - DEBUG - Launching master (ami: ami-4583572c, type: cg1.4xlarge) 2012-05-20 15:16:13,590 PID: 1394 awsutils.py:165 - INFO - Creating security group @sc-sfmpc... 2012-05-20 15:16:18,695 PID: 1394 awsutils.py:275 - INFO - Creating placement group @sc-sfmpc... 2012-05-20 15:16:20,400 PID: 1394 cli.py:257 - ERROR - Unsupported: Your requested instance type (cg1.4xlarge) is not supported in your requested Availability Zone (us-east-1c). Please retry your request by not specifying an Availability Zone or choosing us-east-1a. 2012-05-20 15:17:38,282 PID: 1407 config.py:551 - DEBUG - Loading config 2012-05-20 15:17:38,282 PID: 1407 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 15:17:38,284 PID: 1407 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 15:17:41,692 PID: 1407 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 15:17:41,693 PID: 1407 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 15:17:42,303 PID: 1407 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 15:17:42,303 PID: 1407 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 15:17:42,304 PID: 1407 cli.py:279 - ERROR - Cluster 'sfmcluster' already exists. Either choose a different tag name, or terminate the existing cluster using: $ starcluster terminate sfmcluster If you encountered an issue while starting or using 'sfmcluster' you can reboot and reconfigure the cluster using the 'restart' command: $ starcluster restart sfmcluster This will reboot all existing nodes and completely reconfigure the cluster without wasting instance hours. 2012-05-20 15:17:45,901 PID: 1409 config.py:551 - DEBUG - Loading config 2012-05-20 15:17:45,901 PID: 1409 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 15:17:45,903 PID: 1409 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 15:17:48,587 PID: 1409 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 15:17:48,588 PID: 1409 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 15:17:51,344 PID: 1409 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 15:17:51,344 PID: 1409 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 15:17:51,958 PID: 1409 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 15:17:51,958 PID: 1409 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 15:17:53,834 PID: 1409 cluster.py:1140 - INFO - Waiting for cluster to terminate... 2012-05-20 15:17:54,942 PID: 1409 cluster.py:1377 - INFO - Removing @sc-sfmpc placement group 2012-05-20 15:17:55,550 PID: 1409 cluster.py:1380 - INFO - Removing @sc-sfmpc security group 2012-05-20 15:18:14,093 PID: 1416 config.py:551 - DEBUG - Loading config 2012-05-20 15:18:14,094 PID: 1416 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 15:18:14,096 PID: 1416 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 15:18:15,598 PID: 1416 start.py:176 - INFO - Using default cluster template: sfmcluster 2012-05-20 15:18:15,598 PID: 1416 cluster.py:1539 - INFO - Validating cluster template settings... 2012-05-20 15:18:20,478 PID: 1416 cluster.py:1555 - INFO - Cluster template settings are valid 2012-05-20 15:18:20,478 PID: 1416 cluster.py:1427 - INFO - Starting cluster... 2012-05-20 15:18:20,478 PID: 1416 cluster.py:952 - INFO - Launching a 1-node cluster... 2012-05-20 15:18:20,479 PID: 1416 cluster.py:979 - DEBUG - Launching master (ami: ami-4583572c, type: cg1.4xlarge) 2012-05-20 15:18:21,090 PID: 1416 awsutils.py:165 - INFO - Creating security group @sc-sfmpc... 2012-05-20 15:18:25,718 PID: 1416 cli.py:279 - ERROR - Request to start cluster 'sfmcluster' was cancelled!!! Please be aware that instances may still be running. You can check this from the output of: $ starcluster listclusters If you wish to destroy these instances please run: $ starcluster terminate sfmcluster You can then use: $ starcluster listclusters to verify that the cluster has been terminated. If you would like to re-use these instances, rerun the same start command with the -x (--no-create) option 2012-05-20 15:18:30,658 PID: 1418 config.py:551 - DEBUG - Loading config 2012-05-20 15:18:30,658 PID: 1418 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 15:18:30,659 PID: 1418 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 15:18:33,368 PID: 1418 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 15:18:33,368 PID: 1418 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 15:18:35,281 PID: 1418 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 15:18:35,281 PID: 1418 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 15:18:35,890 PID: 1418 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 15:18:35,891 PID: 1418 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 15:18:37,770 PID: 1418 cluster.py:1140 - INFO - Waiting for cluster to terminate... 2012-05-20 15:18:38,878 PID: 1418 cluster.py:1380 - INFO - Removing @sc-sfmpc security group 2012-05-20 15:18:53,749 PID: 1422 config.py:551 - DEBUG - Loading config 2012-05-20 15:18:53,750 PID: 1422 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 15:18:53,752 PID: 1422 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 15:18:55,233 PID: 1422 start.py:176 - INFO - Using default cluster template: sfmcluster 2012-05-20 15:18:55,234 PID: 1422 cluster.py:1539 - INFO - Validating cluster template settings... 2012-05-20 15:19:00,106 PID: 1422 cluster.py:1555 - INFO - Cluster template settings are valid 2012-05-20 15:19:00,107 PID: 1422 cluster.py:1427 - INFO - Starting cluster... 2012-05-20 15:19:00,107 PID: 1422 cluster.py:952 - INFO - Launching a 1-node cluster... 2012-05-20 15:19:00,107 PID: 1422 cluster.py:1004 - INFO - Launching master node (ami: ami-4583572c, type: cg1.4xlarge)... 2012-05-20 15:19:00,713 PID: 1422 awsutils.py:165 - INFO - Creating security group @sc-sfmpc... 2012-05-20 15:19:06,688 PID: 1422 awsutils.py:275 - INFO - Creating placement group @sc-sfmpc... 2012-05-20 15:19:08,584 PID: 1422 cluster.py:772 - INFO - SpotInstanceRequest:sir-5d9fee11 2012-05-20 15:19:08,584 PID: 1422 utils.py:93 - INFO - Starting cluster took 0.141 mins 2012-05-20 15:19:13,087 PID: 1427 config.py:551 - DEBUG - Loading config 2012-05-20 15:19:13,087 PID: 1427 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 15:19:13,089 PID: 1427 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 15:19:16,470 PID: 1427 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 15:19:16,471 PID: 1427 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 15:19:19,181 PID: 1429 config.py:551 - DEBUG - Loading config 2012-05-20 15:19:19,182 PID: 1429 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 15:19:19,183 PID: 1429 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 15:19:21,885 PID: 1429 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 15:19:21,886 PID: 1429 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 15:20:14,190 PID: 1433 config.py:551 - DEBUG - Loading config 2012-05-20 15:20:14,191 PID: 1433 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 15:20:14,192 PID: 1433 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 15:20:16,900 PID: 1433 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 15:20:16,900 PID: 1433 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 15:20:23,429 PID: 1436 config.py:551 - DEBUG - Loading config 2012-05-20 15:20:23,429 PID: 1436 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 15:20:23,436 PID: 1436 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 15:20:26,274 PID: 1436 awsutils.py:1141 - INFO - Current price: $0.67 2012-05-20 15:20:26,274 PID: 1436 awsutils.py:1142 - INFO - Max price: $2.65 2012-05-20 15:20:26,274 PID: 1436 awsutils.py:1143 - INFO - Average price: $0.75 2012-05-20 15:20:26,275 PID: 1436 awsutils.py:1159 - INFO - 2012-05-20 15:20:26,275 PID: 1436 awsutils.py:1160 - INFO - Starting StarCluster Webserver... 2012-05-20 15:20:26,276 PID: 1436 awsutils.py:1166 - INFO - Server address is http://127.0.0.1:54561 2012-05-20 15:20:26,276 PID: 1436 awsutils.py:1168 - INFO - (use CTRL-C or navigate to http://127.0.0.1:54561/shutdown to shutdown server) 2012-05-20 15:20:26,277 PID: 1436 webtools.py:190 - INFO - Browsing http://127.0.0.1:54561/spothistory.html using 'default'... 2012-05-20 15:20:26,800 PID: 1436 webtools.py:56 - INFO - Shutting down server... 2012-05-20 15:21:29,575 PID: 1441 config.py:551 - DEBUG - Loading config 2012-05-20 15:21:29,575 PID: 1441 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 15:21:29,576 PID: 1441 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 15:21:34,804 PID: 1441 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 15:21:34,805 PID: 1441 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 15:22:08,252 PID: 1449 config.py:551 - DEBUG - Loading config 2012-05-20 15:22:08,252 PID: 1449 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 15:22:08,254 PID: 1449 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 15:22:12,002 PID: 1449 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 15:22:12,002 PID: 1449 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 15:22:14,068 PID: 1449 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 15:22:14,068 PID: 1449 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 15:22:14,718 PID: 1449 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 15:22:14,718 PID: 1449 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 15:22:15,383 PID: 1449 cluster.py:1368 - INFO - Canceling spot instance request: sir-5d9fee11 2012-05-20 15:22:17,830 PID: 1449 cluster.py:1140 - INFO - Waiting for cluster to terminate... 2012-05-20 15:22:19,261 PID: 1449 cluster.py:1377 - INFO - Removing @sc-sfmpc placement group 2012-05-20 15:22:20,052 PID: 1449 cluster.py:1380 - INFO - Removing @sc-sfmpc security group 2012-05-20 15:22:26,832 PID: 1452 config.py:551 - DEBUG - Loading config 2012-05-20 15:22:26,832 PID: 1452 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 15:22:26,835 PID: 1452 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 15:22:28,622 PID: 1452 start.py:176 - INFO - Using default cluster template: sfmcluster 2012-05-20 15:22:28,623 PID: 1452 cluster.py:1539 - INFO - Validating cluster template settings... 2012-05-20 15:22:35,040 PID: 1452 cluster.py:1555 - INFO - Cluster template settings are valid 2012-05-20 15:22:35,041 PID: 1452 cluster.py:1427 - INFO - Starting cluster... 2012-05-20 15:22:35,041 PID: 1452 cluster.py:952 - INFO - Launching a 1-node cluster... 2012-05-20 15:22:35,041 PID: 1452 cluster.py:979 - DEBUG - Launching master (ami: ami-4583572c, type: cg1.4xlarge) 2012-05-20 15:22:35,956 PID: 1452 awsutils.py:165 - INFO - Creating security group @sc-sfmpc... 2012-05-20 15:22:43,948 PID: 1452 awsutils.py:275 - INFO - Creating placement group @sc-sfmpc... 2012-05-20 15:22:46,532 PID: 1452 cli.py:257 - ERROR - Unsupported: Your requested instance type (cg1.4xlarge) is not supported in your requested Availability Zone (us-east-1c). Please retry your request by not specifying an Availability Zone or choosing us-east-1a. 2012-05-20 15:24:52,803 PID: 1460 config.py:551 - DEBUG - Loading config 2012-05-20 15:24:52,803 PID: 1460 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 15:24:52,805 PID: 1460 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 15:24:57,291 PID: 1460 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 15:24:57,291 PID: 1460 cluster.py:672 - DEBUG - adding node i-37551151 to self._nodes list 2012-05-20 15:24:58,257 PID: 1460 cli.py:287 - DEBUG - Traceback (most recent call last): File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/cli.py", line 255, in main sc.execute(args) File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/commands/listclusters.py", line 19, in execute show_ssh_status=self.opts.show_ssh_status) File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/cluster.py", line 247, in list_clusters nodes = cl.nodes File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/cluster.py", line 674, in nodes if n.is_master(): File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/node.py", line 764, in is_master return self.alias == "master" File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/node.py", line 108, in alias user_data = self._get_user_data(tries=5) File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/node.py", line 88, in _get_user_data user_data = self.ec2.get_instance_user_data(self.id) File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/awsutils.py", line 450, in get_instance_user_data return base64.b64decode(user_data) File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/base64.py", line 73, in b64decode return binascii.a2b_base64(s) TypeError: a2b_base64() argument 1 must be string or read-only character buffer, not None 2012-05-20 15:24:58,312 PID: 1460 cli.py:165 - ERROR - Oops! Looks like you've found a bug in StarCluster 2012-05-20 15:24:58,313 PID: 1460 cli.py:166 - ERROR - Crash report written to: /Users/mhelal/.starcluster/logs/crash-report-1460.txt 2012-05-20 15:24:58,313 PID: 1460 cli.py:167 - ERROR - Please remove any sensitive data from the crash report 2012-05-20 15:24:58,313 PID: 1460 cli.py:168 - ERROR - and submit it to starcluster@mit.edu 2012-05-20 15:25:05,157 PID: 1470 config.py:551 - DEBUG - Loading config 2012-05-20 15:25:05,157 PID: 1470 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 15:25:05,159 PID: 1470 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 15:25:08,661 PID: 1470 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 15:25:08,662 PID: 1470 cluster.py:672 - DEBUG - adding node i-37551151 to self._nodes list 2012-05-20 15:25:09,578 PID: 1470 cli.py:287 - DEBUG - Traceback (most recent call last): File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/cli.py", line 255, in main sc.execute(args) File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/commands/listclusters.py", line 19, in execute show_ssh_status=self.opts.show_ssh_status) File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/cluster.py", line 247, in list_clusters nodes = cl.nodes File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/cluster.py", line 674, in nodes if n.is_master(): File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/node.py", line 764, in is_master return self.alias == "master" File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/node.py", line 108, in alias user_data = self._get_user_data(tries=5) File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/node.py", line 88, in _get_user_data user_data = self.ec2.get_instance_user_data(self.id) File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/awsutils.py", line 450, in get_instance_user_data return base64.b64decode(user_data) File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/base64.py", line 73, in b64decode return binascii.a2b_base64(s) TypeError: a2b_base64() argument 1 must be string or read-only character buffer, not None 2012-05-20 15:25:09,591 PID: 1470 cli.py:165 - ERROR - Oops! Looks like you've found a bug in StarCluster 2012-05-20 15:25:09,591 PID: 1470 cli.py:166 - ERROR - Crash report written to: /Users/mhelal/.starcluster/logs/crash-report-1470.txt 2012-05-20 15:25:09,592 PID: 1470 cli.py:167 - ERROR - Please remove any sensitive data from the crash report 2012-05-20 15:25:09,592 PID: 1470 cli.py:168 - ERROR - and submit it to starcluster@mit.edu 2012-05-20 15:25:41,820 PID: 1477 config.py:551 - DEBUG - Loading config 2012-05-20 15:25:41,820 PID: 1477 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 15:25:41,822 PID: 1477 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 15:25:45,511 PID: 1477 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 15:25:45,511 PID: 1477 cluster.py:672 - DEBUG - adding node i-37551151 to self._nodes list 2012-05-20 15:25:46,204 PID: 1477 cli.py:287 - DEBUG - Traceback (most recent call last): File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/cli.py", line 255, in main sc.execute(args) File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/commands/listclusters.py", line 19, in execute show_ssh_status=self.opts.show_ssh_status) File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/cluster.py", line 247, in list_clusters nodes = cl.nodes File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/cluster.py", line 674, in nodes if n.is_master(): File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/node.py", line 764, in is_master return self.alias == "master" File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/node.py", line 108, in alias user_data = self._get_user_data(tries=5) File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/node.py", line 88, in _get_user_data user_data = self.ec2.get_instance_user_data(self.id) File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/awsutils.py", line 450, in get_instance_user_data return base64.b64decode(user_data) File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/base64.py", line 73, in b64decode return binascii.a2b_base64(s) TypeError: a2b_base64() argument 1 must be string or read-only character buffer, not None 2012-05-20 15:25:46,217 PID: 1477 cli.py:165 - ERROR - Oops! Looks like you've found a bug in StarCluster 2012-05-20 15:25:46,217 PID: 1477 cli.py:166 - ERROR - Crash report written to: /Users/mhelal/.starcluster/logs/crash-report-1477.txt 2012-05-20 15:25:46,217 PID: 1477 cli.py:167 - ERROR - Please remove any sensitive data from the crash report 2012-05-20 15:25:46,218 PID: 1477 cli.py:168 - ERROR - and submit it to starcluster@mit.edu 2012-05-20 15:26:00,489 PID: 1484 config.py:551 - DEBUG - Loading config 2012-05-20 15:26:00,489 PID: 1484 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 15:26:00,491 PID: 1484 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 15:26:05,799 PID: 1484 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 15:26:05,799 PID: 1484 cluster.py:672 - DEBUG - adding node i-37551151 to self._nodes list 2012-05-20 15:26:07,023 PID: 1484 cli.py:287 - DEBUG - Traceback (most recent call last): File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/cli.py", line 255, in main sc.execute(args) File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/commands/sshmaster.py", line 36, in execute forward_x11=self.opts.forward_x11) File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/cluster.py", line 115, in ssh_to_master forward_x11=forward_x11) File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/cluster.py", line 1918, in ssh_to_master forward_x11=forward_x11) File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/cluster.py", line 1921, in ssh_to_node node = self.get_node_by_alias(alias) File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/cluster.py", line 704, in get_node_by_alias for node in self.nodes: File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/cluster.py", line 674, in nodes if n.is_master(): File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/node.py", line 764, in is_master return self.alias == "master" File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/node.py", line 108, in alias user_data = self._get_user_data(tries=5) File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/node.py", line 88, in _get_user_data user_data = self.ec2.get_instance_user_data(self.id) File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/awsutils.py", line 450, in get_instance_user_data return base64.b64decode(user_data) File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/base64.py", line 73, in b64decode return binascii.a2b_base64(s) TypeError: a2b_base64() argument 1 must be string or read-only character buffer, not None 2012-05-20 15:26:07,035 PID: 1484 cli.py:165 - ERROR - Oops! Looks like you've found a bug in StarCluster 2012-05-20 15:26:07,035 PID: 1484 cli.py:166 - ERROR - Crash report written to: /Users/mhelal/.starcluster/logs/crash-report-1484.txt 2012-05-20 15:26:07,036 PID: 1484 cli.py:167 - ERROR - Please remove any sensitive data from the crash report 2012-05-20 15:26:07,036 PID: 1484 cli.py:168 - ERROR - and submit it to starcluster@mit.edu 2012-05-20 15:33:14,431 PID: 1527 config.py:551 - DEBUG - Loading config 2012-05-20 15:33:14,432 PID: 1527 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 15:33:14,434 PID: 1527 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1a, 'port': None} 2012-05-20 15:40:42,063 PID: 1566 config.py:551 - DEBUG - Loading config 2012-05-20 15:40:42,063 PID: 1566 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 15:40:42,065 PID: 1566 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1a, 'port': None} 2012-05-20 15:41:30,115 PID: 1578 config.py:551 - DEBUG - Loading config 2012-05-20 15:41:30,115 PID: 1578 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 15:41:30,118 PID: 1578 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1a, 'port': None} 2012-05-20 15:45:54,019 PID: 336 config.py:551 - DEBUG - Loading config 2012-05-20 15:45:54,043 PID: 336 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 15:45:54,057 PID: 336 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1a, 'port': None} 2012-05-20 15:46:26,291 PID: 363 config.py:551 - DEBUG - Loading config 2012-05-20 15:46:26,291 PID: 363 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 15:46:26,293 PID: 363 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1, 'port': None} 2012-05-20 15:46:30,855 PID: 363 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 15:46:30,855 PID: 363 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 15:46:38,360 PID: 366 config.py:551 - DEBUG - Loading config 2012-05-20 15:46:38,361 PID: 366 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 15:46:38,363 PID: 366 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1, 'port': None} 2012-05-20 15:46:41,267 PID: 366 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 15:46:41,267 PID: 366 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 15:46:41,808 PID: 366 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 15:46:41,808 PID: 366 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 15:46:41,809 PID: 366 cli.py:279 - ERROR - Cluster 'sfmcluster' already exists. Either choose a different tag name, or terminate the existing cluster using: $ starcluster terminate sfmcluster If you encountered an issue while starting or using 'sfmcluster' you can reboot and reconfigure the cluster using the 'restart' command: $ starcluster restart sfmcluster This will reboot all existing nodes and completely reconfigure the cluster without wasting instance hours. 2012-05-20 15:46:46,635 PID: 367 config.py:551 - DEBUG - Loading config 2012-05-20 15:46:46,635 PID: 367 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 15:46:46,637 PID: 367 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1, 'port': None} 2012-05-20 15:46:49,496 PID: 367 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 15:46:49,497 PID: 367 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 15:46:51,930 PID: 367 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 15:46:51,931 PID: 367 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 15:46:52,637 PID: 367 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 15:46:52,637 PID: 367 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 15:46:54,477 PID: 367 cluster.py:1140 - INFO - Waiting for cluster to terminate... 2012-05-20 15:46:55,593 PID: 367 cluster.py:1377 - INFO - Removing @sc-sfmpc placement group 2012-05-20 15:46:56,321 PID: 367 cluster.py:1380 - INFO - Removing @sc-sfmpc security group 2012-05-20 15:47:03,770 PID: 371 config.py:551 - DEBUG - Loading config 2012-05-20 15:47:03,770 PID: 371 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 15:47:03,772 PID: 371 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1, 'port': None} 2012-05-20 15:47:05,239 PID: 371 start.py:176 - INFO - Using default cluster template: sfmcluster 2012-05-20 15:47:05,240 PID: 371 cluster.py:1539 - INFO - Validating cluster template settings... 2012-05-20 15:47:10,148 PID: 371 cluster.py:1555 - INFO - Cluster template settings are valid 2012-05-20 15:47:10,149 PID: 371 cluster.py:1427 - INFO - Starting cluster... 2012-05-20 15:47:10,149 PID: 371 cluster.py:952 - INFO - Launching a 1-node cluster... 2012-05-20 15:47:10,149 PID: 371 cluster.py:979 - DEBUG - Launching master (ami: ami-4583572c, type: cg1.4xlarge) 2012-05-20 15:47:10,762 PID: 371 awsutils.py:165 - INFO - Creating security group @sc-sfmpc... 2012-05-20 15:47:16,905 PID: 371 awsutils.py:275 - INFO - Creating placement group @sc-sfmpc... 2012-05-20 15:47:19,514 PID: 371 cli.py:257 - ERROR - Unsupported: Your requested instance type (cg1.4xlarge) is not supported in your requested Availability Zone (us-east-1c). Please retry your request by not specifying an Availability Zone or choosing us-east-1a. 2012-05-20 15:47:31,008 PID: 374 config.py:551 - DEBUG - Loading config 2012-05-20 15:47:31,008 PID: 374 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 15:47:31,010 PID: 374 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1, 'port': None} 2012-05-20 15:47:34,106 PID: 374 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 15:47:34,107 PID: 374 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 15:47:56,225 PID: 374 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 15:47:56,226 PID: 374 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 15:47:56,840 PID: 374 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 15:47:56,841 PID: 374 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 15:47:58,683 PID: 374 cluster.py:1140 - INFO - Waiting for cluster to terminate... 2012-05-20 15:47:59,795 PID: 374 cluster.py:1377 - INFO - Removing @sc-sfmpc placement group 2012-05-20 15:48:00,532 PID: 374 cluster.py:1380 - INFO - Removing @sc-sfmpc security group 2012-05-20 15:57:00,142 PID: 425 config.py:551 - DEBUG - Loading config 2012-05-20 15:57:00,143 PID: 425 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 15:57:00,144 PID: 425 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1, 'port': None} 2012-05-20 15:57:02,532 PID: 425 cluster.py:226 - INFO - No clusters found... 2012-05-20 15:57:12,140 PID: 429 config.py:551 - DEBUG - Loading config 2012-05-20 15:57:12,140 PID: 429 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 15:57:12,141 PID: 429 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1, 'port': None} 2012-05-20 15:57:13,591 PID: 429 cluster.py:226 - INFO - No clusters found... 2012-05-20 15:59:25,043 PID: 440 config.py:551 - DEBUG - Loading config 2012-05-20 15:59:25,043 PID: 440 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 15:59:25,045 PID: 440 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 15:59:27,191 PID: 440 start.py:176 - INFO - Using default cluster template: sfmcluster 2012-05-20 15:59:27,191 PID: 440 cluster.py:1539 - INFO - Validating cluster template settings... 2012-05-20 15:59:29,162 PID: 440 cli.py:279 - ERROR - zone us-east-1 does not exist in region us-east-1 2012-05-20 16:00:00,978 PID: 451 config.py:551 - DEBUG - Loading config 2012-05-20 16:00:00,978 PID: 451 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:00:00,980 PID: 451 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:00:02,634 PID: 451 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:00:04,186 PID: 451 cli.py:272 - ERROR - cluster 'sfmcluster' does not exist 2012-05-20 16:00:06,918 PID: 453 config.py:551 - DEBUG - Loading config 2012-05-20 16:00:06,918 PID: 453 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:00:06,920 PID: 453 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:00:08,489 PID: 453 start.py:176 - INFO - Using default cluster template: sfmcluster 2012-05-20 16:00:08,490 PID: 453 cluster.py:1539 - INFO - Validating cluster template settings... 2012-05-20 16:00:11,876 PID: 453 cli.py:279 - ERROR - Cluster settings are not valid: availability_zone setting 'us-east-1a' does not match the common volume zone 'us-east-1c' 2012-05-20 16:00:35,858 PID: 460 config.py:551 - DEBUG - Loading config 2012-05-20 16:00:35,858 PID: 460 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:00:35,860 PID: 460 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1a, 'port': None} 2012-05-20 16:01:27,688 PID: 465 config.py:551 - DEBUG - Loading config 2012-05-20 16:01:27,688 PID: 465 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:01:27,690 PID: 465 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1a, 'port': None} 2012-05-20 16:04:56,301 PID: 480 config.py:551 - DEBUG - Loading config 2012-05-20 16:04:56,301 PID: 480 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:04:56,303 PID: 480 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1a, 'port': None} 2012-05-20 16:07:24,010 PID: 506 config.py:551 - DEBUG - Loading config 2012-05-20 16:07:24,010 PID: 506 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:07:24,011 PID: 506 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:07:27,598 PID: 506 cluster.py:226 - INFO - No clusters found... 2012-05-20 16:07:37,741 PID: 510 config.py:551 - DEBUG - Loading config 2012-05-20 16:07:37,741 PID: 510 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:07:37,743 PID: 510 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:07:39,253 PID: 510 start.py:176 - INFO - Using default cluster template: sfmcluster 2012-05-20 16:07:39,254 PID: 510 cluster.py:1539 - INFO - Validating cluster template settings... 2012-05-20 16:07:45,087 PID: 510 cluster.py:1555 - INFO - Cluster template settings are valid 2012-05-20 16:07:45,088 PID: 510 cluster.py:1427 - INFO - Starting cluster... 2012-05-20 16:07:45,088 PID: 510 cluster.py:952 - INFO - Launching a 1-node cluster... 2012-05-20 16:07:45,088 PID: 510 cluster.py:979 - DEBUG - Launching master (ami: ami-4583572c, type: cg1.4xlarge) 2012-05-20 16:07:45,698 PID: 510 awsutils.py:165 - INFO - Creating security group @sc-sfmpc... 2012-05-20 16:07:52,152 PID: 510 awsutils.py:275 - INFO - Creating placement group @sc-sfmpc... 2012-05-20 16:07:57,371 PID: 510 cli.py:257 - ERROR - Unsupported: Your requested instance type (cg1.4xlarge) is not supported in your requested Availability Zone (us-east-1c). Please retry your request by not specifying an Availability Zone or choosing us-east-1a. 2012-05-20 16:08:18,346 PID: 515 config.py:551 - DEBUG - Loading config 2012-05-20 16:08:18,346 PID: 515 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:08:18,348 PID: 515 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:08:21,139 PID: 515 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:08:21,139 PID: 515 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:08:22,663 PID: 515 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:08:22,664 PID: 515 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:08:23,278 PID: 515 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:08:23,278 PID: 515 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:08:25,442 PID: 515 cluster.py:1140 - INFO - Waiting for cluster to terminate... 2012-05-20 16:08:26,549 PID: 515 cluster.py:1377 - INFO - Removing @sc-sfmpc placement group 2012-05-20 16:08:27,272 PID: 515 cluster.py:1380 - INFO - Removing @sc-sfmpc security group 2012-05-20 16:10:33,888 PID: 529 config.py:551 - DEBUG - Loading config 2012-05-20 16:10:33,888 PID: 529 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:10:44,463 PID: 530 config.py:551 - DEBUG - Loading config 2012-05-20 16:10:44,463 PID: 530 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:12:32,897 PID: 542 config.py:551 - DEBUG - Loading config 2012-05-20 16:12:32,897 PID: 542 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:12:32,899 PID: 542 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:12:35,510 PID: 542 start.py:176 - INFO - Using default cluster template: sfmcluster 2012-05-20 16:12:35,511 PID: 542 cluster.py:1539 - INFO - Validating cluster template settings... 2012-05-20 16:12:40,407 PID: 542 cluster.py:1555 - INFO - Cluster template settings are valid 2012-05-20 16:12:40,407 PID: 542 cluster.py:1427 - INFO - Starting cluster... 2012-05-20 16:12:40,407 PID: 542 cluster.py:952 - INFO - Launching a 1-node cluster... 2012-05-20 16:12:40,407 PID: 542 cluster.py:1004 - INFO - Launching master node (ami: ami-4583572c, type: cg1.4xlarge)... 2012-05-20 16:12:41,019 PID: 542 awsutils.py:165 - INFO - Creating security group @sc-sfmpc... 2012-05-20 16:12:46,547 PID: 542 awsutils.py:275 - INFO - Creating placement group @sc-sfmpc... 2012-05-20 16:12:49,007 PID: 542 cluster.py:772 - INFO - SpotInstanceRequest:sir-1281c811 2012-05-20 16:12:49,007 PID: 542 utils.py:93 - INFO - Starting cluster took 0.143 mins 2012-05-20 16:12:54,358 PID: 546 config.py:551 - DEBUG - Loading config 2012-05-20 16:12:54,358 PID: 546 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:12:54,359 PID: 546 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:12:57,299 PID: 546 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:12:57,300 PID: 546 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:12:59,659 PID: 550 config.py:551 - DEBUG - Loading config 2012-05-20 16:12:59,659 PID: 550 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:12:59,661 PID: 550 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:13:02,522 PID: 550 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:13:02,522 PID: 550 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:13:04,614 PID: 551 config.py:551 - DEBUG - Loading config 2012-05-20 16:13:04,615 PID: 551 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:13:04,616 PID: 551 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:13:07,438 PID: 551 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:13:07,438 PID: 551 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:13:09,378 PID: 552 config.py:551 - DEBUG - Loading config 2012-05-20 16:13:09,378 PID: 552 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:13:09,380 PID: 552 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:13:12,352 PID: 552 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:13:12,353 PID: 552 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:13:14,414 PID: 554 config.py:551 - DEBUG - Loading config 2012-05-20 16:13:14,414 PID: 554 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:13:14,416 PID: 554 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:13:17,567 PID: 554 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:13:17,567 PID: 554 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:13:22,814 PID: 555 config.py:551 - DEBUG - Loading config 2012-05-20 16:13:22,814 PID: 555 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:13:22,816 PID: 555 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:13:25,563 PID: 555 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:13:25,563 PID: 555 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:13:27,712 PID: 555 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:13:27,713 PID: 555 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:13:28,328 PID: 555 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:13:28,328 PID: 555 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:13:29,009 PID: 555 cluster.py:1368 - INFO - Canceling spot instance request: sir-1281c811 2012-05-20 16:13:31,327 PID: 555 cluster.py:1140 - INFO - Waiting for cluster to terminate... 2012-05-20 16:13:32,518 PID: 555 cluster.py:1377 - INFO - Removing @sc-sfmpc placement group 2012-05-20 16:13:33,321 PID: 555 cluster.py:1380 - INFO - Removing @sc-sfmpc security group 2012-05-20 16:13:44,796 PID: 557 config.py:551 - DEBUG - Loading config 2012-05-20 16:13:44,796 PID: 557 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:13:44,798 PID: 557 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:13:46,768 PID: 557 start.py:176 - INFO - Using default cluster template: sfmcluster 2012-05-20 16:13:46,768 PID: 557 cluster.py:1539 - INFO - Validating cluster template settings... 2012-05-20 16:13:51,677 PID: 557 cluster.py:1555 - INFO - Cluster template settings are valid 2012-05-20 16:13:51,677 PID: 557 cluster.py:1427 - INFO - Starting cluster... 2012-05-20 16:13:51,678 PID: 557 cluster.py:952 - INFO - Launching a 1-node cluster... 2012-05-20 16:13:51,678 PID: 557 cluster.py:1004 - INFO - Launching master node (ami: ami-4583572c, type: cg1.4xlarge)... 2012-05-20 16:13:52,289 PID: 557 awsutils.py:165 - INFO - Creating security group @sc-sfmpc... 2012-05-20 16:13:58,131 PID: 557 awsutils.py:275 - INFO - Creating placement group @sc-sfmpc... 2012-05-20 16:14:00,175 PID: 557 cluster.py:772 - INFO - SpotInstanceRequest:sir-b6ccce11 2012-05-20 16:14:00,176 PID: 557 utils.py:93 - INFO - Starting cluster took 0.142 mins 2012-05-20 16:14:02,683 PID: 559 config.py:551 - DEBUG - Loading config 2012-05-20 16:14:02,683 PID: 559 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:14:02,685 PID: 559 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:14:07,982 PID: 559 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:14:07,982 PID: 559 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:14:10,545 PID: 560 config.py:551 - DEBUG - Loading config 2012-05-20 16:14:10,545 PID: 560 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:14:10,547 PID: 560 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:14:15,046 PID: 560 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:14:15,047 PID: 560 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:14:17,316 PID: 563 config.py:551 - DEBUG - Loading config 2012-05-20 16:14:17,316 PID: 563 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:14:17,318 PID: 563 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:14:21,268 PID: 563 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:14:21,268 PID: 563 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:14:23,101 PID: 564 config.py:551 - DEBUG - Loading config 2012-05-20 16:14:23,101 PID: 564 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:14:23,102 PID: 564 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:14:26,215 PID: 564 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:14:26,215 PID: 564 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:14:28,853 PID: 565 config.py:551 - DEBUG - Loading config 2012-05-20 16:14:28,853 PID: 565 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:14:28,855 PID: 565 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:14:31,764 PID: 565 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:14:31,764 PID: 565 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:14:33,763 PID: 567 config.py:551 - DEBUG - Loading config 2012-05-20 16:14:33,763 PID: 567 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:14:33,765 PID: 567 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:14:37,576 PID: 567 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:14:37,577 PID: 567 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:14:50,088 PID: 568 config.py:551 - DEBUG - Loading config 2012-05-20 16:14:50,088 PID: 568 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:14:50,090 PID: 568 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:14:53,549 PID: 568 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:14:53,549 PID: 568 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:14:55,701 PID: 568 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:14:55,702 PID: 568 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:14:56,314 PID: 568 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:14:56,314 PID: 568 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:14:57,253 PID: 568 cluster.py:1368 - INFO - Canceling spot instance request: sir-b6ccce11 2012-05-20 16:14:59,670 PID: 568 cluster.py:1140 - INFO - Waiting for cluster to terminate... 2012-05-20 16:15:02,064 PID: 568 cluster.py:1377 - INFO - Removing @sc-sfmpc placement group 2012-05-20 16:15:02,795 PID: 568 cluster.py:1380 - INFO - Removing @sc-sfmpc security group 2012-05-20 16:15:14,183 PID: 575 config.py:551 - DEBUG - Loading config 2012-05-20 16:15:14,184 PID: 575 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:15:14,186 PID: 575 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:15:15,955 PID: 575 start.py:176 - INFO - Using default cluster template: sfmcluster 2012-05-20 16:15:15,956 PID: 575 cluster.py:1539 - INFO - Validating cluster template settings... 2012-05-20 16:15:24,248 PID: 575 cluster.py:1555 - INFO - Cluster template settings are valid 2012-05-20 16:15:24,248 PID: 575 cluster.py:1427 - INFO - Starting cluster... 2012-05-20 16:15:24,248 PID: 575 cluster.py:952 - INFO - Launching a 1-node cluster... 2012-05-20 16:15:24,249 PID: 575 cluster.py:979 - DEBUG - Launching master (ami: ami-4583572c, type: cg1.4xlarge) 2012-05-20 16:15:24,890 PID: 575 awsutils.py:165 - INFO - Creating security group @sc-sfmpc... 2012-05-20 16:15:32,879 PID: 575 awsutils.py:275 - INFO - Creating placement group @sc-sfmpc... 2012-05-20 16:15:35,431 PID: 575 cli.py:257 - ERROR - Unsupported: Your requested instance type (cg1.4xlarge) is not supported in your requested Availability Zone (us-east-1c). Please retry your request by not specifying an Availability Zone or choosing us-east-1a. 2012-05-20 16:19:04,105 PID: 590 config.py:551 - DEBUG - Loading config 2012-05-20 16:19:04,105 PID: 590 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:19:04,107 PID: 590 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:19:06,763 PID: 590 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:sa-east-1, 'port': None} 2012-05-20 16:19:09,944 PID: 590 awsutils.py:656 - INFO - Listing all public StarCluster images... 2012-05-20 16:19:31,556 PID: 595 config.py:551 - DEBUG - Loading config 2012-05-20 16:19:31,556 PID: 595 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:19:31,557 PID: 595 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:19:33,171 PID: 595 cli.py:279 - ERROR - region us-east-1c does not exist 2012-05-20 16:19:39,815 PID: 596 config.py:551 - DEBUG - Loading config 2012-05-20 16:19:39,815 PID: 596 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:19:39,817 PID: 596 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:19:41,775 PID: 596 cli.py:279 - ERROR - region us-east-1a does not exist 2012-05-20 16:20:26,370 PID: 599 config.py:551 - DEBUG - Loading config 2012-05-20 16:20:26,370 PID: 599 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:20:26,371 PID: 599 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:20:28,667 PID: 599 awsutils.py:656 - INFO - Listing all public StarCluster images... 2012-05-20 16:21:32,775 PID: 610 config.py:551 - DEBUG - Loading config 2012-05-20 16:21:32,776 PID: 610 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:21:32,778 PID: 610 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:21:37,095 PID: 610 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:21:37,096 PID: 610 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:21:37,695 PID: 610 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:21:37,695 PID: 610 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:21:37,696 PID: 610 cli.py:279 - ERROR - Cluster 'sfmcluster' already exists. Either choose a different tag name, or terminate the existing cluster using: $ starcluster terminate sfmcluster If you encountered an issue while starting or using 'sfmcluster' you can reboot and reconfigure the cluster using the 'restart' command: $ starcluster restart sfmcluster This will reboot all existing nodes and completely reconfigure the cluster without wasting instance hours. 2012-05-20 16:21:43,654 PID: 612 config.py:551 - DEBUG - Loading config 2012-05-20 16:21:43,654 PID: 612 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:21:43,655 PID: 612 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:21:47,221 PID: 612 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:21:47,221 PID: 612 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:21:52,347 PID: 616 config.py:551 - DEBUG - Loading config 2012-05-20 16:21:52,347 PID: 616 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:21:52,349 PID: 616 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:21:55,205 PID: 616 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:21:55,206 PID: 616 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:21:57,360 PID: 616 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:21:57,360 PID: 616 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:21:57,970 PID: 616 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:21:57,970 PID: 616 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:21:59,816 PID: 616 cluster.py:1140 - INFO - Waiting for cluster to terminate... 2012-05-20 16:22:00,929 PID: 616 cluster.py:1377 - INFO - Removing @sc-sfmpc placement group 2012-05-20 16:22:01,656 PID: 616 cluster.py:1380 - INFO - Removing @sc-sfmpc security group 2012-05-20 16:22:06,128 PID: 617 config.py:551 - DEBUG - Loading config 2012-05-20 16:22:06,128 PID: 617 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:22:06,130 PID: 617 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:22:07,633 PID: 617 start.py:176 - INFO - Using default cluster template: sfmcluster 2012-05-20 16:22:07,634 PID: 617 cluster.py:1539 - INFO - Validating cluster template settings... 2012-05-20 16:22:12,718 PID: 617 cluster.py:1555 - INFO - Cluster template settings are valid 2012-05-20 16:22:12,719 PID: 617 cluster.py:1427 - INFO - Starting cluster... 2012-05-20 16:22:12,719 PID: 617 cluster.py:952 - INFO - Launching a 1-node cluster... 2012-05-20 16:22:12,719 PID: 617 cluster.py:979 - DEBUG - Launching master (ami: ami-4583572c, type: cg1.4xlarge) 2012-05-20 16:22:13,330 PID: 617 awsutils.py:165 - INFO - Creating security group @sc-sfmpc... 2012-05-20 16:22:19,167 PID: 617 awsutils.py:275 - INFO - Creating placement group @sc-sfmpc... 2012-05-20 16:22:21,144 PID: 617 cli.py:257 - ERROR - Unsupported: Your requested instance type (cg1.4xlarge) is not supported in your requested Availability Zone (us-east-1c). Please retry your request by not specifying an Availability Zone or choosing us-east-1a. 2012-05-20 16:22:32,561 PID: 622 config.py:551 - DEBUG - Loading config 2012-05-20 16:22:32,561 PID: 622 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:22:32,563 PID: 622 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:22:35,447 PID: 622 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:22:35,447 PID: 622 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:22:36,064 PID: 622 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:22:36,064 PID: 622 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:22:36,065 PID: 622 cli.py:279 - ERROR - Cluster 'sfmcluster' already exists. Either choose a different tag name, or terminate the existing cluster using: $ starcluster terminate sfmcluster If you encountered an issue while starting or using 'sfmcluster' you can reboot and reconfigure the cluster using the 'restart' command: $ starcluster restart sfmcluster This will reboot all existing nodes and completely reconfigure the cluster without wasting instance hours. 2012-05-20 16:22:38,302 PID: 623 config.py:551 - DEBUG - Loading config 2012-05-20 16:22:38,302 PID: 623 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:22:38,304 PID: 623 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:22:40,978 PID: 623 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:22:40,979 PID: 623 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:22:42,821 PID: 623 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:22:42,821 PID: 623 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:22:44,382 PID: 623 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:22:44,383 PID: 623 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:22:46,202 PID: 623 cluster.py:1140 - INFO - Waiting for cluster to terminate... 2012-05-20 16:22:47,316 PID: 623 cluster.py:1377 - INFO - Removing @sc-sfmpc placement group 2012-05-20 16:22:48,047 PID: 623 cluster.py:1380 - INFO - Removing @sc-sfmpc security group 2012-05-20 16:22:59,992 PID: 625 config.py:551 - DEBUG - Loading config 2012-05-20 16:22:59,993 PID: 625 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:22:59,995 PID: 625 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:23:04,639 PID: 625 start.py:176 - INFO - Using default cluster template: sfmcluster 2012-05-20 16:23:04,640 PID: 625 cluster.py:1539 - INFO - Validating cluster template settings... 2012-05-20 16:23:11,105 PID: 625 cluster.py:1555 - INFO - Cluster template settings are valid 2012-05-20 16:23:11,106 PID: 625 cluster.py:1427 - INFO - Starting cluster... 2012-05-20 16:23:11,106 PID: 625 cluster.py:952 - INFO - Launching a 1-node cluster... 2012-05-20 16:23:11,106 PID: 625 cluster.py:979 - DEBUG - Launching master (ami: ami-4583572c, type: cg1.4xlarge) 2012-05-20 16:23:11,730 PID: 625 awsutils.py:165 - INFO - Creating security group @sc-sfmpc... 2012-05-20 16:23:19,714 PID: 625 awsutils.py:275 - INFO - Creating placement group @sc-sfmpc... 2012-05-20 16:23:22,619 PID: 625 cli.py:257 - ERROR - Unsupported: Your requested instance type (cg1.4xlarge) is not supported in your requested Availability Zone (us-east-1c). Please retry your request by not specifying an Availability Zone or choosing us-east-1a. 2012-05-20 16:23:32,555 PID: 630 config.py:551 - DEBUG - Loading config 2012-05-20 16:23:32,555 PID: 630 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:23:32,557 PID: 630 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:23:35,691 PID: 630 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:23:35,692 PID: 630 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:23:47,663 PID: 630 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:23:47,664 PID: 630 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:23:48,586 PID: 630 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:23:48,586 PID: 630 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:23:51,541 PID: 630 cluster.py:1140 - INFO - Waiting for cluster to terminate... 2012-05-20 16:23:52,967 PID: 630 cluster.py:1377 - INFO - Removing @sc-sfmpc placement group 2012-05-20 16:23:53,889 PID: 630 cluster.py:1380 - INFO - Removing @sc-sfmpc security group 2012-05-20 16:24:12,257 PID: 642 config.py:551 - DEBUG - Loading config 2012-05-20 16:24:12,257 PID: 642 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:24:12,259 PID: 642 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1, 'port': None} 2012-05-20 16:24:14,377 PID: 642 start.py:176 - INFO - Using default cluster template: sfmcluster 2012-05-20 16:24:14,378 PID: 642 cluster.py:1539 - INFO - Validating cluster template settings... 2012-05-20 16:24:20,309 PID: 642 cluster.py:1555 - INFO - Cluster template settings are valid 2012-05-20 16:24:20,309 PID: 642 cluster.py:1427 - INFO - Starting cluster... 2012-05-20 16:24:20,309 PID: 642 cluster.py:952 - INFO - Launching a 1-node cluster... 2012-05-20 16:24:20,310 PID: 642 cluster.py:979 - DEBUG - Launching master (ami: ami-4583572c, type: cg1.4xlarge) 2012-05-20 16:24:20,921 PID: 642 awsutils.py:165 - INFO - Creating security group @sc-sfmpc... 2012-05-20 16:24:26,761 PID: 642 awsutils.py:275 - INFO - Creating placement group @sc-sfmpc... 2012-05-20 16:24:29,030 PID: 642 cli.py:257 - ERROR - Unsupported: Your requested instance type (cg1.4xlarge) is not supported in your requested Availability Zone (us-east-1c). Please retry your request by not specifying an Availability Zone or choosing us-east-1a. 2012-05-20 16:24:58,966 PID: 651 config.py:551 - DEBUG - Loading config 2012-05-20 16:24:58,966 PID: 651 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:24:58,967 PID: 651 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:25:02,083 PID: 651 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:25:02,083 PID: 651 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:25:04,542 PID: 651 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:25:04,542 PID: 651 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:25:05,157 PID: 651 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:25:05,158 PID: 651 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:25:07,306 PID: 651 cluster.py:1140 - INFO - Waiting for cluster to terminate... 2012-05-20 16:25:08,524 PID: 651 cluster.py:1377 - INFO - Removing @sc-sfmpc placement group 2012-05-20 16:25:09,253 PID: 651 cluster.py:1380 - INFO - Removing @sc-sfmpc security group 2012-05-20 16:25:12,850 PID: 654 config.py:551 - DEBUG - Loading config 2012-05-20 16:25:12,851 PID: 654 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:25:12,853 PID: 654 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:25:14,498 PID: 654 start.py:176 - INFO - Using default cluster template: sfmcluster 2012-05-20 16:25:14,499 PID: 654 cluster.py:1539 - INFO - Validating cluster template settings... 2012-05-20 16:25:19,707 PID: 654 cluster.py:1555 - INFO - Cluster template settings are valid 2012-05-20 16:25:19,707 PID: 654 cluster.py:1427 - INFO - Starting cluster... 2012-05-20 16:25:19,708 PID: 654 cluster.py:952 - INFO - Launching a 1-node cluster... 2012-05-20 16:25:19,708 PID: 654 cluster.py:979 - DEBUG - Launching master (ami: ami-4583572c, type: cg1.4xlarge) 2012-05-20 16:25:20,315 PID: 654 awsutils.py:165 - INFO - Creating security group @sc-sfmpc... 2012-05-20 16:25:27,686 PID: 654 awsutils.py:275 - INFO - Creating placement group @sc-sfmpc... 2012-05-20 16:25:29,628 PID: 654 cli.py:257 - ERROR - Unsupported: Your requested instance type (cg1.4xlarge) is not supported in your requested Availability Zone (us-east-1c). Please retry your request by not specifying an Availability Zone or choosing us-east-1a. 2012-05-20 16:29:51,855 PID: 675 config.py:551 - DEBUG - Loading config 2012-05-20 16:29:51,855 PID: 675 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:29:51,856 PID: 675 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:29:55,666 PID: 675 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:29:55,666 PID: 675 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:29:58,154 PID: 678 config.py:551 - DEBUG - Loading config 2012-05-20 16:29:58,154 PID: 678 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:29:58,155 PID: 678 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:30:00,892 PID: 678 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:30:00,892 PID: 678 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 16:30:04,158 PID: 683 config.py:551 - DEBUG - Loading config 2012-05-20 16:30:04,159 PID: 683 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 16:30:04,160 PID: 683 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 16:30:07,966 PID: 683 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 16:30:07,966 PID: 683 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 17:39:50,388 PID: 1036 config.py:551 - DEBUG - Loading config 2012-05-20 17:39:50,389 PID: 1036 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 17:39:50,391 PID: 1036 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 17:39:56,122 PID: 1036 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 17:39:56,123 PID: 1036 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 17:44:43,430 PID: 1075 config.py:551 - DEBUG - Loading config 2012-05-20 17:44:43,430 PID: 1075 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 17:44:43,432 PID: 1075 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 17:44:48,706 PID: 1075 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 17:44:48,707 PID: 1075 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 17:44:49,578 PID: 1075 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 17:44:49,578 PID: 1075 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 17:44:49,578 PID: 1075 cli.py:279 - ERROR - Cluster 'sfmcluster' already exists. Either choose a different tag name, or terminate the existing cluster using: $ starcluster terminate sfmcluster If you encountered an issue while starting or using 'sfmcluster' you can reboot and reconfigure the cluster using the 'restart' command: $ starcluster restart sfmcluster This will reboot all existing nodes and completely reconfigure the cluster without wasting instance hours. 2012-05-20 17:44:57,161 PID: 1079 config.py:551 - DEBUG - Loading config 2012-05-20 17:44:57,161 PID: 1079 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 17:44:57,163 PID: 1079 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 17:45:00,672 PID: 1079 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 17:45:00,673 PID: 1079 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 17:45:03,155 PID: 1079 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 17:45:03,155 PID: 1079 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 17:45:04,016 PID: 1079 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 17:45:04,017 PID: 1079 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 17:45:06,065 PID: 1079 cluster.py:1140 - INFO - Waiting for cluster to terminate... 2012-05-20 17:45:07,210 PID: 1079 cluster.py:1377 - INFO - Removing @sc-sfmpc placement group 2012-05-20 17:45:08,627 PID: 1079 cluster.py:1380 - INFO - Removing @sc-sfmpc security group 2012-05-20 17:45:13,059 PID: 1087 config.py:551 - DEBUG - Loading config 2012-05-20 17:45:13,059 PID: 1087 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 17:45:13,061 PID: 1087 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 17:45:15,248 PID: 1087 start.py:176 - INFO - Using default cluster template: sfmcluster 2012-05-20 17:45:15,249 PID: 1087 cluster.py:1539 - INFO - Validating cluster template settings... 2012-05-20 17:45:22,861 PID: 1087 cluster.py:1555 - INFO - Cluster template settings are valid 2012-05-20 17:45:22,861 PID: 1087 cluster.py:1427 - INFO - Starting cluster... 2012-05-20 17:45:22,862 PID: 1087 cluster.py:952 - INFO - Launching a 1-node cluster... 2012-05-20 17:45:22,862 PID: 1087 cluster.py:979 - DEBUG - Launching master (ami: ami-4583572c, type: cg1.4xlarge) 2012-05-20 17:45:23,506 PID: 1087 awsutils.py:165 - INFO - Creating security group @sc-sfmpc... 2012-05-20 17:45:31,821 PID: 1087 awsutils.py:275 - INFO - Creating placement group @sc-sfmpc... 2012-05-20 17:45:34,687 PID: 1087 cli.py:257 - ERROR - Unsupported: Your requested instance type (cg1.4xlarge) is not supported in your requested Availability Zone (us-east-1c). Please retry your request by not specifying an Availability Zone or choosing us-east-1a. 2012-05-20 17:45:37,675 PID: 1089 config.py:551 - DEBUG - Loading config 2012-05-20 17:45:37,675 PID: 1089 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 17:45:37,677 PID: 1089 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 17:45:41,318 PID: 1089 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 17:45:41,318 PID: 1089 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 17:45:43,790 PID: 1089 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 17:45:43,791 PID: 1089 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 17:45:44,701 PID: 1089 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 17:45:44,702 PID: 1089 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 17:45:47,447 PID: 1089 cluster.py:1140 - INFO - Waiting for cluster to terminate... 2012-05-20 17:45:48,905 PID: 1089 cluster.py:1377 - INFO - Removing @sc-sfmpc placement group 2012-05-20 17:45:49,321 PID: 1089 cluster.py:1380 - INFO - Removing @sc-sfmpc security group 2012-05-20 17:51:23,248 PID: 1128 config.py:551 - DEBUG - Loading config 2012-05-20 17:51:23,248 PID: 1128 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 17:51:23,250 PID: 1128 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 17:52:35,411 PID: 1135 config.py:551 - DEBUG - Loading config 2012-05-20 17:52:35,412 PID: 1135 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 17:52:35,413 PID: 1135 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 17:52:37,553 PID: 1135 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 17:52:40,043 PID: 1135 cli.py:272 - ERROR - cluster 'sfmcluster' does not exist 2012-05-20 17:52:42,030 PID: 1138 config.py:551 - DEBUG - Loading config 2012-05-20 17:52:42,030 PID: 1138 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 17:52:42,032 PID: 1138 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 17:52:43,555 PID: 1138 start.py:176 - INFO - Using default cluster template: sfmcluster 2012-05-20 17:52:43,555 PID: 1138 cluster.py:1539 - INFO - Validating cluster template settings... 2012-05-20 17:52:46,157 PID: 1138 cli.py:279 - ERROR - Cluster settings are not valid: availability_zone setting 'us-east-1a' does not match the common volume zone 'us-east-1c' 2012-05-20 17:55:35,189 PID: 1150 config.py:551 - DEBUG - Loading config 2012-05-20 17:55:35,189 PID: 1150 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 17:55:35,192 PID: 1150 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1, 'port': None} 2012-05-20 17:55:37,401 PID: 1150 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1, 'port': None} 2012-05-20 17:55:38,867 PID: 1150 cli.py:272 - ERROR - cluster 'sfmcluster' does not exist 2012-05-20 17:55:41,484 PID: 1152 config.py:551 - DEBUG - Loading config 2012-05-20 17:55:41,484 PID: 1152 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 17:55:41,486 PID: 1152 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1, 'port': None} 2012-05-20 17:55:42,947 PID: 1152 start.py:176 - INFO - Using default cluster template: sfmcluster 2012-05-20 17:55:42,948 PID: 1152 cluster.py:1539 - INFO - Validating cluster template settings... 2012-05-20 17:55:45,421 PID: 1152 cli.py:279 - ERROR - zone us-east-1 does not exist in region us-east-1 2012-05-20 17:56:15,824 PID: 1159 config.py:551 - DEBUG - Loading config 2012-05-20 17:56:15,824 PID: 1159 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 17:56:15,826 PID: 1159 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1, 'port': None} 2012-05-20 17:56:17,289 PID: 1159 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1, 'port': None} 2012-05-20 17:56:18,724 PID: 1159 cli.py:272 - ERROR - cluster 'sfmcluster' does not exist 2012-05-20 17:56:20,133 PID: 1160 config.py:551 - DEBUG - Loading config 2012-05-20 17:56:20,133 PID: 1160 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 17:56:20,135 PID: 1160 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1, 'port': None} 2012-05-20 17:56:21,588 PID: 1160 start.py:176 - INFO - Using default cluster template: sfmcluster 2012-05-20 17:56:21,589 PID: 1160 cluster.py:1539 - INFO - Validating cluster template settings... 2012-05-20 17:56:24,021 PID: 1160 cli.py:279 - ERROR - Cluster settings are not valid: availability_zone setting 'us-east-1a' does not match the common volume zone 'us-east-1c' 2012-05-20 17:57:01,902 PID: 1172 config.py:551 - DEBUG - Loading config 2012-05-20 17:57:01,903 PID: 1172 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 17:57:01,904 PID: 1172 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1, 'port': None} 2012-05-20 17:57:04,288 PID: 1172 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1, 'port': None} 2012-05-20 17:57:05,788 PID: 1172 cli.py:272 - ERROR - cluster 'sfmcluster' does not exist 2012-05-20 17:57:07,501 PID: 1174 config.py:551 - DEBUG - Loading config 2012-05-20 17:57:07,501 PID: 1174 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 17:57:07,503 PID: 1174 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1, 'port': None} 2012-05-20 17:57:09,060 PID: 1174 start.py:176 - INFO - Using default cluster template: sfmcluster 2012-05-20 17:57:09,060 PID: 1174 cluster.py:1539 - INFO - Validating cluster template settings... 2012-05-20 17:57:14,807 PID: 1174 cluster.py:1555 - INFO - Cluster template settings are valid 2012-05-20 17:57:14,807 PID: 1174 cluster.py:1427 - INFO - Starting cluster... 2012-05-20 17:57:14,807 PID: 1174 cluster.py:952 - INFO - Launching a 1-node cluster... 2012-05-20 17:57:14,808 PID: 1174 cluster.py:979 - DEBUG - Launching master (ami: ami-4583572c, type: cg1.4xlarge) 2012-05-20 17:57:15,448 PID: 1174 awsutils.py:165 - INFO - Creating security group @sc-sfmpc... 2012-05-20 17:57:21,546 PID: 1174 awsutils.py:275 - INFO - Creating placement group @sc-sfmpc... 2012-05-20 17:57:24,234 PID: 1174 cli.py:257 - ERROR - Unsupported: Your requested instance type (cg1.4xlarge) is not supported in your requested Availability Zone (us-east-1c). Please retry your request by not specifying an Availability Zone or choosing us-east-1a. 2012-05-20 17:57:37,994 PID: 1181 config.py:551 - DEBUG - Loading config 2012-05-20 17:57:37,994 PID: 1181 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 17:57:37,996 PID: 1181 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1a, 'port': None} 2012-05-20 17:57:40,867 PID: 1181 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 17:57:40,867 PID: 1181 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 17:57:42,738 PID: 1181 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 17:57:42,738 PID: 1181 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 17:57:43,363 PID: 1181 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 17:57:43,363 PID: 1181 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 17:57:44,673 PID: 1181 awsutils.py:307 - DEBUG - region us-east-1a not in CLUSTER_REGIONS (us-east-1) 2012-05-20 17:57:44,673 PID: 1181 cluster.py:1140 - INFO - Waiting for cluster to terminate... 2012-05-20 17:57:45,814 PID: 1181 cluster.py:1380 - INFO - Removing @sc-sfmpc security group 2012-05-20 18:15:36,840 PID: 1263 config.py:551 - DEBUG - Loading config 2012-05-20 18:15:36,840 PID: 1263 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 18:15:36,842 PID: 1263 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 18:15:39,705 PID: 1263 start.py:176 - INFO - Using default cluster template: sfmcluster 2012-05-20 18:15:39,706 PID: 1263 cluster.py:1539 - INFO - Validating cluster template settings... 2012-05-20 18:15:46,298 PID: 1263 cluster.py:1555 - INFO - Cluster template settings are valid 2012-05-20 18:15:46,298 PID: 1263 cluster.py:1427 - INFO - Starting cluster... 2012-05-20 18:15:46,299 PID: 1263 cluster.py:952 - INFO - Launching a 1-node cluster... 2012-05-20 18:15:46,299 PID: 1263 cluster.py:979 - DEBUG - Launching master (ami: ami-4583572c, type: cg1.4xlarge) 2012-05-20 18:15:46,924 PID: 1263 awsutils.py:165 - INFO - Creating security group @sc-sfmpc... 2012-05-20 18:15:55,949 PID: 1263 cli.py:257 - ERROR - Unsupported: Your requested instance type (cg1.4xlarge) is not supported in your requested Availability Zone (us-east-1c). Please retry your request by not specifying an Availability Zone or choosing us-east-1a. 2012-05-20 18:16:10,744 PID: 1271 config.py:551 - DEBUG - Loading config 2012-05-20 18:16:10,745 PID: 1271 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 18:16:10,746 PID: 1271 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1a, 'port': None} 2012-05-20 18:17:46,055 PID: 1284 config.py:551 - DEBUG - Loading config 2012-05-20 18:17:46,055 PID: 1284 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 18:17:46,057 PID: 1284 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1a, 'port': None} 2012-05-20 18:17:50,820 PID: 1284 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 18:17:50,820 PID: 1284 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 18:17:54,508 PID: 1284 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 18:17:54,509 PID: 1284 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 18:17:55,428 PID: 1284 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 18:17:55,428 PID: 1284 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 18:17:57,266 PID: 1284 awsutils.py:307 - DEBUG - region us-east-1a not in CLUSTER_REGIONS (us-east-1) 2012-05-20 18:17:57,266 PID: 1284 cluster.py:1140 - INFO - Waiting for cluster to terminate... 2012-05-20 18:17:58,690 PID: 1284 cluster.py:1380 - INFO - Removing @sc-sfmpc security group 2012-05-20 18:18:04,934 PID: 1287 config.py:551 - DEBUG - Loading config 2012-05-20 18:18:04,934 PID: 1287 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 18:18:04,936 PID: 1287 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1a, 'port': None} 2012-05-20 18:18:06,503 PID: 1287 start.py:176 - INFO - Using default cluster template: sfmcluster 2012-05-20 18:18:06,504 PID: 1287 cluster.py:1539 - INFO - Validating cluster template settings... 2012-05-20 18:18:13,247 PID: 1287 cluster.py:1555 - INFO - Cluster template settings are valid 2012-05-20 18:18:13,247 PID: 1287 cluster.py:1427 - INFO - Starting cluster... 2012-05-20 18:18:13,248 PID: 1287 cluster.py:952 - INFO - Launching a 1-node cluster... 2012-05-20 18:18:13,248 PID: 1287 cluster.py:979 - DEBUG - Launching master (ami: ami-4583572c, type: cg1.4xlarge) 2012-05-20 18:18:13,844 PID: 1287 awsutils.py:165 - INFO - Creating security group @sc-sfmpc... 2012-05-20 18:18:23,005 PID: 1287 cli.py:257 - ERROR - Unsupported: Your requested instance type (cg1.4xlarge) is not supported in your requested Availability Zone (us-east-1c). Please retry your request by not specifying an Availability Zone or choosing us-east-1a. 2012-05-20 19:26:54,822 PID: 1624 config.py:551 - DEBUG - Loading config 2012-05-20 19:26:54,822 PID: 1624 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 19:26:54,824 PID: 1624 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1a, 'port': None} 2012-05-20 19:27:01,068 PID: 1624 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 19:27:01,069 PID: 1624 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 19:27:03,540 PID: 1624 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 19:27:03,541 PID: 1624 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 19:27:04,607 PID: 1624 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 19:27:04,607 PID: 1624 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 19:27:05,987 PID: 1624 awsutils.py:307 - DEBUG - region us-east-1a not in CLUSTER_REGIONS (us-east-1) 2012-05-20 19:27:05,988 PID: 1624 cluster.py:1140 - INFO - Waiting for cluster to terminate... 2012-05-20 19:27:07,409 PID: 1624 cluster.py:1380 - INFO - Removing @sc-sfmpc security group 2012-05-20 19:27:13,426 PID: 1628 config.py:551 - DEBUG - Loading config 2012-05-20 19:27:13,426 PID: 1628 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 19:27:13,428 PID: 1628 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1a, 'port': None} 2012-05-20 19:27:15,228 PID: 1628 start.py:176 - INFO - Using default cluster template: sfmcluster 2012-05-20 19:27:15,229 PID: 1628 cluster.py:1539 - INFO - Validating cluster template settings... 2012-05-20 19:27:22,875 PID: 1628 cluster.py:1555 - INFO - Cluster template settings are valid 2012-05-20 19:27:22,875 PID: 1628 cluster.py:1427 - INFO - Starting cluster... 2012-05-20 19:27:22,876 PID: 1628 cluster.py:952 - INFO - Launching a 1-node cluster... 2012-05-20 19:27:22,876 PID: 1628 cluster.py:979 - DEBUG - Launching master (ami: ami-4583572c, type: cg1.4xlarge) 2012-05-20 19:27:23,801 PID: 1628 awsutils.py:165 - INFO - Creating security group @sc-sfmpc... 2012-05-20 19:27:32,514 PID: 1628 cli.py:257 - ERROR - Unsupported: Your requested instance type (cg1.4xlarge) is not supported in your requested Availability Zone (us-east-1c). Please retry your request by not specifying an Availability Zone or choosing us-east-1a. 2012-05-20 19:27:42,950 PID: 1632 config.py:551 - DEBUG - Loading config 2012-05-20 19:27:42,950 PID: 1632 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 19:27:42,951 PID: 1632 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1a, 'port': None} 2012-05-20 19:27:46,537 PID: 1632 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 19:27:46,538 PID: 1632 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 19:27:51,456 PID: 1632 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 19:27:51,457 PID: 1632 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 19:27:52,387 PID: 1632 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 19:27:52,388 PID: 1632 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 19:27:54,227 PID: 1632 awsutils.py:307 - DEBUG - region us-east-1a not in CLUSTER_REGIONS (us-east-1) 2012-05-20 19:27:54,227 PID: 1632 cluster.py:1140 - INFO - Waiting for cluster to terminate... 2012-05-20 19:27:56,448 PID: 1632 cluster.py:1380 - INFO - Removing @sc-sfmpc security group 2012-05-20 20:41:10,151 PID: 1955 config.py:551 - DEBUG - Loading config 2012-05-20 20:41:10,151 PID: 1955 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 20:41:10,153 PID: 1955 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1a, 'port': None} 2012-05-20 20:41:13,045 PID: 1955 start.py:176 - INFO - Using default cluster template: sfmcluster 2012-05-20 20:41:13,046 PID: 1955 cluster.py:1539 - INFO - Validating cluster template settings... 2012-05-20 20:41:20,961 PID: 1955 cluster.py:1555 - INFO - Cluster template settings are valid 2012-05-20 20:41:20,961 PID: 1955 cluster.py:1427 - INFO - Starting cluster... 2012-05-20 20:41:20,961 PID: 1955 cluster.py:952 - INFO - Launching a 1-node cluster... 2012-05-20 20:41:20,962 PID: 1955 cluster.py:979 - DEBUG - Launching master (ami: ami-4583572c, type: cg1.4xlarge) 2012-05-20 20:41:21,592 PID: 1955 awsutils.py:165 - INFO - Creating security group @sc-sfmpc... 2012-05-20 20:41:30,001 PID: 1955 cli.py:257 - ERROR - Unsupported: Your requested instance type (cg1.4xlarge) is not supported in your requested Availability Zone (us-east-1c). Please retry your request by not specifying an Availability Zone or choosing us-east-1a. 2012-05-20 20:41:33,631 PID: 1960 config.py:551 - DEBUG - Loading config 2012-05-20 20:41:33,631 PID: 1960 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 20:41:33,633 PID: 1960 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': RegionInfo:us-east-1a, 'port': None} 2012-05-20 20:41:37,284 PID: 1960 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 20:41:37,284 PID: 1960 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 20:41:39,119 PID: 1960 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 20:41:39,119 PID: 1960 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 20:41:40,050 PID: 1960 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-20 20:41:40,050 PID: 1960 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-20 20:41:42,190 PID: 1960 awsutils.py:307 - DEBUG - region us-east-1a not in CLUSTER_REGIONS (us-east-1) 2012-05-20 20:41:42,190 PID: 1960 cluster.py:1140 - INFO - Waiting for cluster to terminate... 2012-05-20 20:41:43,619 PID: 1960 cluster.py:1380 - INFO - Removing @sc-sfmpc security group 2012-05-20 21:07:17,218 PID: 2076 config.py:551 - DEBUG - Loading config 2012-05-20 21:07:17,218 PID: 2076 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 21:11:58,616 PID: 2102 config.py:551 - DEBUG - Loading config 2012-05-20 21:11:58,616 PID: 2102 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-20 21:11:58,618 PID: 2102 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-20 21:12:01,596 PID: 2102 start.py:176 - INFO - Using default cluster template: sfmcluster 2012-05-20 21:12:01,597 PID: 2102 cluster.py:1539 - INFO - Validating cluster template settings... 2012-05-20 21:12:11,420 PID: 2102 cli.py:279 - ERROR - Cluster settings are not valid: availability_zone setting 'us-east-1a' does not match the common volume zone 'us-east-1c' 2012-05-21 19:21:10,922 PID: 528 config.py:551 - DEBUG - Loading config 2012-05-21 19:21:10,958 PID: 528 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-21 19:21:11,012 PID: 528 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-21 19:21:15,102 PID: 528 awsutils.py:1141 - INFO - Current price: $0.67 2012-05-21 19:21:15,103 PID: 528 awsutils.py:1142 - INFO - Max price: $2.65 2012-05-21 19:21:15,103 PID: 528 awsutils.py:1143 - INFO - Average price: $0.75 2012-05-21 19:21:15,103 PID: 528 awsutils.py:1159 - INFO - 2012-05-21 19:21:15,103 PID: 528 awsutils.py:1160 - INFO - Starting StarCluster Webserver... 2012-05-21 19:21:15,105 PID: 528 awsutils.py:1166 - INFO - Server address is http://127.0.0.1:49964 2012-05-21 19:21:15,105 PID: 528 awsutils.py:1168 - INFO - (use CTRL-C or navigate to http://127.0.0.1:49964/shutdown to shutdown server) 2012-05-21 19:21:15,106 PID: 528 webtools.py:190 - INFO - Browsing http://127.0.0.1:49964/spothistory.html using 'default'... 2012-05-21 19:21:15,992 PID: 528 webtools.py:56 - INFO - Shutting down server... 2012-05-21 19:21:27,139 PID: 535 config.py:551 - DEBUG - Loading config 2012-05-21 19:21:27,139 PID: 535 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-21 19:21:27,141 PID: 535 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-21 19:21:28,851 PID: 535 start.py:176 - INFO - Using default cluster template: sfmcluster 2012-05-21 19:21:28,852 PID: 535 cluster.py:1539 - INFO - Validating cluster template settings... 2012-05-21 19:21:33,982 PID: 535 cluster.py:1555 - INFO - Cluster template settings are valid 2012-05-21 19:21:33,983 PID: 535 cluster.py:1427 - INFO - Starting cluster... 2012-05-21 19:21:33,983 PID: 535 cluster.py:952 - INFO - Launching a 1-node cluster... 2012-05-21 19:21:33,983 PID: 535 cluster.py:1004 - INFO - Launching master node (ami: ami-4583572c, type: cg1.4xlarge)... 2012-05-21 19:21:34,561 PID: 535 awsutils.py:165 - INFO - Creating security group @sc-sfmpc... 2012-05-21 19:21:41,662 PID: 535 cluster.py:772 - INFO - SpotInstanceRequest:sir-9a91ec11 2012-05-21 19:21:41,663 PID: 535 utils.py:93 - INFO - Starting cluster took 0.128 mins 2012-05-21 19:21:50,975 PID: 537 config.py:551 - DEBUG - Loading config 2012-05-21 19:21:50,976 PID: 537 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-21 19:21:50,977 PID: 537 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-21 19:21:53,647 PID: 537 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-21 19:21:53,647 PID: 537 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-21 19:22:35,908 PID: 543 config.py:551 - DEBUG - Loading config 2012-05-21 19:22:35,908 PID: 543 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-21 19:22:35,910 PID: 543 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-21 19:22:39,843 PID: 543 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-21 19:22:39,843 PID: 543 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-21 19:23:33,373 PID: 548 config.py:551 - DEBUG - Loading config 2012-05-21 19:23:33,373 PID: 548 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-21 19:23:33,374 PID: 548 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-21 19:23:36,358 PID: 548 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-21 19:23:36,358 PID: 548 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-21 19:25:01,818 PID: 555 config.py:551 - DEBUG - Loading config 2012-05-21 19:25:01,818 PID: 555 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-21 19:25:01,820 PID: 555 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-21 19:25:05,335 PID: 555 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-21 19:25:05,335 PID: 555 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-21 19:25:38,108 PID: 563 config.py:551 - DEBUG - Loading config 2012-05-21 19:25:38,109 PID: 563 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-21 19:25:38,110 PID: 563 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-21 19:25:40,858 PID: 563 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-21 19:25:40,858 PID: 563 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-21 19:30:23,423 PID: 585 config.py:551 - DEBUG - Loading config 2012-05-21 19:30:23,423 PID: 585 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-21 19:30:23,425 PID: 585 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-21 19:30:27,786 PID: 585 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-21 19:30:27,786 PID: 585 cluster.py:672 - DEBUG - adding node i-474a0521 to self._nodes list 2012-05-21 19:30:29,841 PID: 585 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-21 19:30:31,176 PID: 585 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-21 19:30:31,177 PID: 585 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-21 19:30:31,178 PID: 585 __init__.py:97 - DEBUG - connecting to host ec2-23-22-0-208.compute-1.amazonaws.com on port 22 as user root 2012-05-21 19:30:37,688 PID: 585 __init__.py:186 - DEBUG - creating sftp connection 2012-05-21 19:30:45,349 PID: 589 config.py:551 - DEBUG - Loading config 2012-05-21 19:30:45,349 PID: 589 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-21 19:30:45,351 PID: 589 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-21 19:30:50,151 PID: 589 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-21 19:30:50,151 PID: 589 cluster.py:672 - DEBUG - adding node i-474a0521 to self._nodes list 2012-05-21 19:30:50,152 PID: 589 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-21 19:30:50,846 PID: 589 node.py:910 - DEBUG - Using native OpenSSH client 2012-05-21 19:30:50,846 PID: 589 node.py:919 - DEBUG - ssh_cmd: ssh -i /Users/mhelal/.ssh/mhelal.rsa root@ec2-23-22-0-208.compute-1.amazonaws.com 2012-05-21 19:35:29,771 PID: 623 config.py:551 - DEBUG - Loading config 2012-05-21 19:35:29,771 PID: 623 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-21 19:35:29,773 PID: 623 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-21 19:35:33,989 PID: 623 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-21 19:35:33,989 PID: 623 cluster.py:672 - DEBUG - adding node i-474a0521 to self._nodes list 2012-05-21 19:35:33,989 PID: 623 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-21 19:35:36,421 PID: 623 cluster.py:664 - DEBUG - existing nodes: {u'i-474a0521': } 2012-05-21 19:35:36,421 PID: 623 cluster.py:667 - DEBUG - updating existing node i-474a0521 in self._nodes 2012-05-21 19:35:36,421 PID: 623 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-21 19:35:37,050 PID: 623 cluster.py:664 - DEBUG - existing nodes: {u'i-474a0521': } 2012-05-21 19:35:37,050 PID: 623 cluster.py:667 - DEBUG - updating existing node i-474a0521 in self._nodes 2012-05-21 19:35:37,050 PID: 623 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-21 19:35:37,050 PID: 623 node.py:828 - INFO - Terminating node: master (i-474a0521) 2012-05-21 19:35:38,379 PID: 623 cluster.py:1368 - INFO - Canceling spot instance request: sir-9a91ec11 2012-05-21 19:35:40,205 PID: 623 cluster.py:1140 - INFO - Waiting for cluster to terminate... 2012-05-21 19:37:29,311 PID: 623 cluster.py:1377 - INFO - Removing @sc-sfmpc placement group 2012-05-21 19:37:29,981 PID: 623 cluster.py:1380 - INFO - Removing @sc-sfmpc security group 2012-05-21 19:39:04,102 PID: 644 config.py:551 - DEBUG - Loading config 2012-05-21 19:39:04,102 PID: 644 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-21 19:39:04,104 PID: 644 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-21 19:39:06,596 PID: 644 start.py:176 - INFO - Using default cluster template: sfmcluster 2012-05-21 19:39:06,597 PID: 644 cluster.py:1539 - INFO - Validating cluster template settings... 2012-05-21 19:39:11,663 PID: 644 cluster.py:1555 - INFO - Cluster template settings are valid 2012-05-21 19:39:11,663 PID: 644 cluster.py:1427 - INFO - Starting cluster... 2012-05-21 19:39:11,664 PID: 644 cluster.py:952 - INFO - Launching a 1-node cluster... 2012-05-21 19:39:11,664 PID: 644 cluster.py:1004 - INFO - Launching master node (ami: ami-4583572c, type: cg1.4xlarge)... 2012-05-21 19:39:12,288 PID: 644 awsutils.py:165 - INFO - Creating security group @sc-sfmpc... 2012-05-21 19:39:18,522 PID: 644 awsutils.py:275 - INFO - Creating placement group @sc-sfmpc... 2012-05-21 19:39:20,786 PID: 644 cluster.py:772 - INFO - SpotInstanceRequest:sir-7b0c8414 2012-05-21 19:39:20,787 PID: 644 utils.py:93 - INFO - Starting cluster took 0.152 mins 2012-05-21 19:39:43,053 PID: 647 config.py:551 - DEBUG - Loading config 2012-05-21 19:39:43,053 PID: 647 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-21 19:39:43,054 PID: 647 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-21 19:39:46,083 PID: 647 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-21 19:39:46,083 PID: 647 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-21 19:40:55,767 PID: 657 config.py:551 - DEBUG - Loading config 2012-05-21 19:40:55,767 PID: 657 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-21 19:40:55,769 PID: 657 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-21 19:40:59,509 PID: 657 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-21 19:40:59,510 PID: 657 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-21 19:42:13,701 PID: 664 config.py:551 - DEBUG - Loading config 2012-05-21 19:42:13,702 PID: 664 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-21 19:42:13,703 PID: 664 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-21 19:42:17,215 PID: 664 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-21 19:42:17,215 PID: 664 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-21 19:45:34,635 PID: 679 config.py:551 - DEBUG - Loading config 2012-05-21 19:45:34,635 PID: 679 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-21 19:45:34,636 PID: 679 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-21 19:45:38,669 PID: 679 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-21 19:45:38,670 PID: 679 cluster.py:672 - DEBUG - adding node i-156e2173 to self._nodes list 2012-05-21 19:45:42,097 PID: 679 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-21 19:45:43,692 PID: 679 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-21 19:45:43,693 PID: 679 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-21 19:45:43,694 PID: 679 __init__.py:97 - DEBUG - connecting to host ec2-204-236-203-128.compute-1.amazonaws.com on port 22 as user root 2012-05-21 19:46:17,808 PID: 683 config.py:551 - DEBUG - Loading config 2012-05-21 19:46:17,808 PID: 683 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-21 19:46:17,810 PID: 683 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-21 19:46:22,063 PID: 683 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-21 19:46:22,064 PID: 683 cluster.py:672 - DEBUG - adding node i-156e2173 to self._nodes list 2012-05-21 19:46:22,064 PID: 683 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-21 19:46:22,678 PID: 683 node.py:910 - DEBUG - Using native OpenSSH client 2012-05-21 19:46:22,678 PID: 683 node.py:919 - DEBUG - ssh_cmd: ssh -i /Users/mhelal/.ssh/mhelal.rsa root@ec2-204-236-203-128.compute-1.amazonaws.com 2012-05-21 19:47:06,902 PID: 693 config.py:551 - DEBUG - Loading config 2012-05-21 19:47:06,902 PID: 693 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-21 19:47:06,904 PID: 693 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-21 19:47:11,949 PID: 693 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-21 19:47:11,949 PID: 693 cluster.py:672 - DEBUG - adding node i-156e2173 to self._nodes list 2012-05-21 19:47:11,949 PID: 693 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-21 19:47:12,664 PID: 693 node.py:910 - DEBUG - Using native OpenSSH client 2012-05-21 19:47:12,664 PID: 693 node.py:919 - DEBUG - ssh_cmd: ssh -i /Users/mhelal/.ssh/mhelal.rsa root@ec2-204-236-203-128.compute-1.amazonaws.com 2012-05-21 19:49:15,635 PID: 703 config.py:551 - DEBUG - Loading config 2012-05-21 19:49:15,636 PID: 703 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-21 19:49:15,637 PID: 703 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-21 19:49:21,186 PID: 703 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-21 19:49:21,186 PID: 703 cluster.py:672 - DEBUG - adding node i-156e2173 to self._nodes list 2012-05-21 19:49:21,187 PID: 703 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-21 19:49:21,187 PID: 703 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-21 19:49:21,188 PID: 703 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-21 19:49:21,188 PID: 703 __init__.py:194 - DEBUG - creating scp connection 2012-05-21 19:49:21,189 PID: 703 __init__.py:97 - DEBUG - connecting to host ec2-204-236-203-128.compute-1.amazonaws.com on port 22 as user root 2012-05-21 19:49:28,722 PID: 703 __init__.py:186 - DEBUG - creating sftp connection 2012-05-21 19:50:35,032 PID: 713 config.py:551 - DEBUG - Loading config 2012-05-21 19:50:35,032 PID: 713 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-21 19:50:35,034 PID: 713 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-21 19:50:40,320 PID: 713 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-21 19:50:40,320 PID: 713 cluster.py:672 - DEBUG - adding node i-156e2173 to self._nodes list 2012-05-21 19:50:40,320 PID: 713 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-21 19:50:40,320 PID: 713 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-21 19:50:40,322 PID: 713 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-21 19:50:40,322 PID: 713 __init__.py:194 - DEBUG - creating scp connection 2012-05-21 19:50:40,322 PID: 713 __init__.py:97 - DEBUG - connecting to host ec2-204-236-203-128.compute-1.amazonaws.com on port 22 as user root 2012-05-21 19:50:46,247 PID: 713 __init__.py:186 - DEBUG - creating sftp connection 2012-05-21 19:53:21,227 PID: 727 config.py:551 - DEBUG - Loading config 2012-05-21 19:53:21,227 PID: 727 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-21 19:53:21,229 PID: 727 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-21 19:53:25,780 PID: 727 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-21 19:53:25,780 PID: 727 cluster.py:672 - DEBUG - adding node i-156e2173 to self._nodes list 2012-05-21 19:53:25,781 PID: 727 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-21 19:53:25,781 PID: 727 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-21 19:53:25,782 PID: 727 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-21 19:53:25,782 PID: 727 __init__.py:194 - DEBUG - creating scp connection 2012-05-21 19:53:25,782 PID: 727 __init__.py:97 - DEBUG - connecting to host ec2-204-236-203-128.compute-1.amazonaws.com on port 22 as user root 2012-05-21 19:53:31,535 PID: 727 __init__.py:186 - DEBUG - creating sftp connection 2012-05-21 19:57:48,514 PID: 769 config.py:551 - DEBUG - Loading config 2012-05-21 19:57:48,514 PID: 769 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-21 19:57:48,515 PID: 769 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-21 19:57:53,193 PID: 769 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-21 19:57:53,193 PID: 769 cluster.py:672 - DEBUG - adding node i-156e2173 to self._nodes list 2012-05-21 19:57:53,193 PID: 769 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-21 19:57:53,193 PID: 769 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-21 19:57:53,194 PID: 769 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-21 19:57:53,194 PID: 769 __init__.py:194 - DEBUG - creating scp connection 2012-05-21 19:57:53,194 PID: 769 __init__.py:97 - DEBUG - connecting to host ec2-204-236-203-128.compute-1.amazonaws.com on port 22 as user root 2012-05-21 19:57:58,991 PID: 769 __init__.py:186 - DEBUG - creating sftp connection 2012-05-21 20:29:11,789 PID: 945 config.py:551 - DEBUG - Loading config 2012-05-21 20:29:11,789 PID: 945 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-21 20:29:11,790 PID: 945 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-21 20:29:16,062 PID: 945 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-21 20:29:16,062 PID: 945 cluster.py:672 - DEBUG - adding node i-156e2173 to self._nodes list 2012-05-21 20:29:16,062 PID: 945 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-21 20:29:16,728 PID: 945 cluster.py:664 - DEBUG - existing nodes: {u'i-156e2173': } 2012-05-21 20:29:16,728 PID: 945 cluster.py:667 - DEBUG - updating existing node i-156e2173 in self._nodes 2012-05-21 20:29:16,729 PID: 945 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-21 20:29:17,440 PID: 945 cluster.py:664 - DEBUG - existing nodes: {u'i-156e2173': } 2012-05-21 20:29:17,440 PID: 945 cluster.py:667 - DEBUG - updating existing node i-156e2173 in self._nodes 2012-05-21 20:29:17,440 PID: 945 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-21 20:29:17,441 PID: 945 cli.py:279 - ERROR - Cluster 'sfmcluster' does not contain any 'stoppable' nodes and can only be terminated. Please use the 'terminate' command instead to destroy the cluster. Pass --help for more info 2012-05-21 20:35:07,208 PID: 977 config.py:551 - DEBUG - Loading config 2012-05-21 20:35:07,209 PID: 977 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-21 20:35:07,210 PID: 977 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-21 20:35:11,100 PID: 977 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-21 20:35:11,100 PID: 977 cluster.py:672 - DEBUG - adding node i-156e2173 to self._nodes list 2012-05-21 20:35:11,100 PID: 977 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-21 20:37:04,991 PID: 977 cluster.py:664 - DEBUG - existing nodes: {u'i-156e2173': } 2012-05-21 20:37:04,991 PID: 977 cluster.py:667 - DEBUG - updating existing node i-156e2173 in self._nodes 2012-05-21 20:37:04,991 PID: 977 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-21 20:37:05,599 PID: 977 cluster.py:664 - DEBUG - existing nodes: {u'i-156e2173': } 2012-05-21 20:37:05,599 PID: 977 cluster.py:667 - DEBUG - updating existing node i-156e2173 in self._nodes 2012-05-21 20:37:05,599 PID: 977 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-21 20:37:05,599 PID: 977 node.py:828 - INFO - Terminating node: master (i-156e2173) 2012-05-21 20:37:06,950 PID: 977 cluster.py:1368 - INFO - Canceling spot instance request: sir-7b0c8414 2012-05-21 20:37:08,960 PID: 977 cluster.py:1140 - INFO - Waiting for cluster to terminate... 2012-05-21 20:38:52,306 PID: 977 cluster.py:1377 - INFO - Removing @sc-sfmpc placement group 2012-05-21 20:38:52,903 PID: 977 cluster.py:1380 - INFO - Removing @sc-sfmpc security group 2012-05-22 16:56:28,690 PID: 1142 config.py:551 - DEBUG - Loading config 2012-05-22 16:56:28,690 PID: 1142 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 16:56:28,692 PID: 1142 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 17:06:08,955 PID: 1189 config.py:551 - DEBUG - Loading config 2012-05-22 17:06:08,956 PID: 1189 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 17:06:08,957 PID: 1189 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 17:06:11,586 PID: 1189 createvolume.py:77 - INFO - No keypair specified, picking one from config... 2012-05-22 17:06:12,277 PID: 1189 createvolume.py:83 - INFO - Using keypair: mhelal 2012-05-22 17:06:13,529 PID: 1189 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 17:06:13,529 PID: 1189 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 17:06:15,309 PID: 1189 awsutils.py:165 - INFO - Creating security group @sc-volumecreator... 2012-05-22 17:06:22,682 PID: 1189 volume.py:77 - INFO - No instance in group @sc-volumecreator for zone us-east-1a, launching one now. 2012-05-22 17:06:23,913 PID: 1189 cluster.py:772 - INFO - Reservation:r-0d155b6f 2012-05-22 17:06:23,913 PID: 1189 cluster.py:1235 - INFO - Waiting for volume host to come up... (updating every 30s) 2012-05-22 17:06:25,818 PID: 1189 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 17:06:25,818 PID: 1189 cluster.py:672 - DEBUG - adding node i-47a5f221 to self._nodes list 2012-05-22 17:06:29,745 PID: 1189 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 17:06:29,745 PID: 1189 cluster.py:1193 - INFO - Waiting for all nodes to be in a 'running' state... 2012-05-22 17:06:30,419 PID: 1189 cluster.py:664 - DEBUG - existing nodes: {u'i-47a5f221': } 2012-05-22 17:06:30,419 PID: 1189 cluster.py:667 - DEBUG - updating existing node i-47a5f221 in self._nodes 2012-05-22 17:06:30,420 PID: 1189 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 17:07:01,161 PID: 1189 cluster.py:664 - DEBUG - existing nodes: {u'i-47a5f221': } 2012-05-22 17:07:01,162 PID: 1189 cluster.py:667 - DEBUG - updating existing node i-47a5f221 in self._nodes 2012-05-22 17:07:01,162 PID: 1189 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 17:07:01,163 PID: 1189 cluster.py:1211 - INFO - Waiting for SSH to come up on all nodes... 2012-05-22 17:07:02,062 PID: 1189 cluster.py:664 - DEBUG - existing nodes: {u'i-47a5f221': } 2012-05-22 17:07:02,062 PID: 1189 cluster.py:667 - DEBUG - updating existing node i-47a5f221 in self._nodes 2012-05-22 17:07:02,062 PID: 1189 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 17:07:02,978 PID: 1189 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-22 17:07:02,994 PID: 1189 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-22 17:07:02,995 PID: 1189 __init__.py:97 - DEBUG - connecting to host ec2-50-16-89-103.compute-1.amazonaws.com on port 22 as user root 2012-05-22 17:07:40,541 PID: 1189 cluster.py:664 - DEBUG - existing nodes: {u'i-47a5f221': } 2012-05-22 17:07:40,541 PID: 1189 cluster.py:667 - DEBUG - updating existing node i-47a5f221 in self._nodes 2012-05-22 17:07:40,542 PID: 1189 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 17:07:41,776 PID: 1189 __init__.py:97 - DEBUG - connecting to host ec2-50-16-89-103.compute-1.amazonaws.com on port 22 as user root 2012-05-22 17:07:47,891 PID: 1189 __init__.py:186 - DEBUG - creating sftp connection 2012-05-22 17:07:49,103 PID: 1189 utils.py:93 - INFO - Waiting for cluster to come up took 1.420 mins 2012-05-22 17:07:50,730 PID: 1189 cluster.py:664 - DEBUG - existing nodes: {u'i-47a5f221': } 2012-05-22 17:07:50,730 PID: 1189 cluster.py:667 - DEBUG - updating existing node i-47a5f221 in self._nodes 2012-05-22 17:07:50,731 PID: 1189 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 17:07:50,731 PID: 1189 volume.py:187 - INFO - Checking for required remote commands... 2012-05-22 17:07:52,075 PID: 1189 __init__.py:543 - DEBUG - /sbin/mkfs.ext3 2012-05-22 17:07:52,076 PID: 1189 volume.py:95 - INFO - Creating 30GB volume in zone us-east-1a 2012-05-22 17:07:53,444 PID: 1189 volume.py:97 - INFO - New volume id: vol-69bd4807 2012-05-22 17:07:53,444 PID: 1189 cluster.py:1140 - INFO - Waiting for new volume to become 'available'... 2012-05-22 17:08:00,505 PID: 1189 cluster.py:1140 - INFO - Attaching volume vol-69bd4807 to instance i-47a5f221... 2012-05-22 17:08:10,098 PID: 1189 volume.py:208 - INFO - Formatting volume... 2012-05-22 17:08:26,169 PID: 1189 __init__.py:543 - DEBUG - Filesystem label= 2012-05-22 17:08:26,170 PID: 1189 __init__.py:543 - DEBUG - OS type: Linux 2012-05-22 17:08:26,171 PID: 1189 __init__.py:543 - DEBUG - Block size=4096 (log=2) 2012-05-22 17:08:26,172 PID: 1189 __init__.py:543 - DEBUG - Fragment size=4096 (log=2) 2012-05-22 17:08:26,172 PID: 1189 __init__.py:543 - DEBUG - Stride=0 blocks, Stripe width=0 blocks 2012-05-22 17:08:26,172 PID: 1189 __init__.py:543 - DEBUG - 1966080 inodes, 7864320 blocks 2012-05-22 17:08:26,173 PID: 1189 __init__.py:543 - DEBUG - 393216 blocks (5.00%) reserved for the super user 2012-05-22 17:08:26,173 PID: 1189 __init__.py:543 - DEBUG - First data block=0 2012-05-22 17:08:26,173 PID: 1189 __init__.py:543 - DEBUG - Maximum filesystem blocks=0 2012-05-22 17:08:26,173 PID: 1189 __init__.py:543 - DEBUG - 240 block groups 2012-05-22 17:08:26,174 PID: 1189 __init__.py:543 - DEBUG - 32768 blocks per group, 32768 fragments per group 2012-05-22 17:08:26,174 PID: 1189 __init__.py:543 - DEBUG - 8192 inodes per group 2012-05-22 17:08:26,174 PID: 1189 __init__.py:543 - DEBUG - Superblock backups stored on blocks: 2012-05-22 17:08:26,174 PID: 1189 __init__.py:543 - DEBUG - 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 2012-05-22 17:08:26,175 PID: 1189 __init__.py:543 - DEBUG - 4096000 2012-05-22 17:08:26,175 PID: 1189 __init__.py:543 - DEBUG - 2012-05-22 17:08:26,175 PID: 1189 __init__.py:543 - DEBUG - Writing inode tables: 0/240 1/240 2/240 3/240 4/240 5/240 6/240 7/240 8/240 9/240 10/240 11/240 12/240 13/240 14/240 15/240 16/240 17/240 18/240 19/240 20/240 21/240 22/240 23/240 24/240 25/240 26/240 27/240 28/240 29/240 30/240 31/240 32/240 33/240 34/240 35/240 36/240 37/240 38/240 39/240 40/240 41/240 42/240 43/240 44/240 45/240 46/240 47/240 48/240 49/240 50/240 51/240 52/240 53/240 54/240 55/240 56/240 57/240 58/240 59/240 60/240 61/240 62/240 63/240 64/240 65/240 66/240 67/240 68/240 69/240 70/240 71/240 72/240 73/240 74/240 75/240 76/240 77/240 78/240 79/240 80/240 81/240 82/240 83/240 84/240 85/240 86/240 87/240 88/240 89/240 90/240 91/240 92/240 93/240 94/240 95/240 96/240 97/240 98/240 99/240100/240101/240102/240103/240104/240105/240106/240107/240108/240109/240110/240111/240112/240113/240114/240115/240116/240117/240118/240119/240120/240121/240122/240123/240124/240125/240126/240127/240128/240129/240130/240131/240132/240133/240134/240135/240136/240137/240138/240139/240140/240141/240142/240143/240144/240145/240146/240147/240148/240149/240150/240151/240152/240153/240154/240155/240156/240157/240158/240159/240160/240161/240162/240163/240164/240165/240166/240167/240168/240169/240170/240171/240172/240173/240174/240175/240176/240177/240178/240179/240180/240181/240182/240183/240184/240185/240186/240187/240188/240189/240190/240191/240192/240193/240194/240195/240196/240197/240198/240199/240200/240201/240202/240203/240204/240205/240206/240207/240208/240209/240210/240211/240212/240213/240214/240215/240216/240217/240218/240219/240220/240221/240222/240223/240224/240225/240226/240227/240228/240229/240230/240231/240232/240233/240234/240235/240236/240237/240238/240239/240done 2012-05-22 17:08:26,176 PID: 1189 __init__.py:543 - DEBUG - Creating journal (32768 blocks): done 2012-05-22 17:08:26,176 PID: 1189 __init__.py:543 - DEBUG - Writing superblocks and filesystem accounting information: done 2012-05-22 17:08:26,176 PID: 1189 __init__.py:543 - DEBUG - 2012-05-22 17:08:26,176 PID: 1189 __init__.py:543 - DEBUG - This filesystem will be automatically checked every 26 mounts or 2012-05-22 17:08:26,177 PID: 1189 __init__.py:543 - DEBUG - 180 days, whichever comes first. Use tune2fs -c or -i to override. 2012-05-22 17:08:26,177 PID: 1189 __init__.py:543 - DEBUG - mke2fs 1.41.14 (22-Dec-2010) 2012-05-22 17:08:26,177 PID: 1189 volume.py:240 - INFO - Leaving volume vol-69bd4807 attached to instance i-47a5f221 2012-05-22 17:08:26,177 PID: 1189 volume.py:246 - INFO - Not terminating host instance i-47a5f221 2012-05-22 17:08:28,125 PID: 1189 volume.py:222 - WARNING - There are still volume hosts running: i-47a5f221 2012-05-22 17:08:28,126 PID: 1189 volume.py:225 - WARNING - Run 'starcluster terminate volumecreator' to terminate *all* volume host instances once they're no longer needed 2012-05-22 17:08:28,126 PID: 1189 volume.py:273 - INFO - Your new 30GB volume vol-69bd4807 has been created successfully 2012-05-22 17:08:28,126 PID: 1189 utils.py:93 - INFO - Creating volume took 2.264 mins 2012-05-22 17:08:41,380 PID: 1203 config.py:551 - DEBUG - Loading config 2012-05-22 17:08:41,381 PID: 1203 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 17:08:41,382 PID: 1203 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 17:08:45,986 PID: 1203 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 17:08:45,986 PID: 1203 cluster.py:672 - DEBUG - adding node i-47a5f221 to self._nodes list 2012-05-22 17:08:45,987 PID: 1203 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 17:11:51,137 PID: 1219 config.py:551 - DEBUG - Loading config 2012-05-22 17:11:51,138 PID: 1219 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 17:11:51,140 PID: 1219 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 17:11:59,476 PID: 1219 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 17:11:59,477 PID: 1219 cluster.py:672 - DEBUG - adding node i-47a5f221 to self._nodes list 2012-05-22 17:11:59,477 PID: 1219 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 17:12:07,162 PID: 1219 cluster.py:664 - DEBUG - existing nodes: {u'i-47a5f221': } 2012-05-22 17:12:07,162 PID: 1219 cluster.py:667 - DEBUG - updating existing node i-47a5f221 in self._nodes 2012-05-22 17:12:07,163 PID: 1219 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 17:12:07,835 PID: 1219 node.py:733 - INFO - Detaching volume vol-69bd4807 from volhost-us-east-1a 2012-05-22 17:12:09,674 PID: 1219 cluster.py:664 - DEBUG - existing nodes: {u'i-47a5f221': } 2012-05-22 17:12:09,675 PID: 1219 cluster.py:667 - DEBUG - updating existing node i-47a5f221 in self._nodes 2012-05-22 17:12:09,675 PID: 1219 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 17:12:09,675 PID: 1219 node.py:828 - INFO - Terminating node: volhost-us-east-1a (i-47a5f221) 2012-05-22 17:12:12,697 PID: 1219 cluster.py:1140 - INFO - Waiting for cluster to terminate... 2012-05-22 17:13:04,062 PID: 1219 cluster.py:1380 - INFO - Removing @sc-volumecreator security group 2012-05-22 17:13:43,724 PID: 1238 config.py:551 - DEBUG - Loading config 2012-05-22 17:13:43,724 PID: 1238 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 17:13:43,732 PID: 1238 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 17:13:48,342 PID: 1238 awsutils.py:1141 - INFO - Current price: $0.67 2012-05-22 17:13:48,343 PID: 1238 awsutils.py:1142 - INFO - Max price: $2.65 2012-05-22 17:13:48,343 PID: 1238 awsutils.py:1143 - INFO - Average price: $0.75 2012-05-22 17:13:48,344 PID: 1238 awsutils.py:1159 - INFO - 2012-05-22 17:13:48,344 PID: 1238 awsutils.py:1160 - INFO - Starting StarCluster Webserver... 2012-05-22 17:13:48,345 PID: 1238 awsutils.py:1166 - INFO - Server address is http://127.0.0.1:51508 2012-05-22 17:13:48,345 PID: 1238 awsutils.py:1168 - INFO - (use CTRL-C or navigate to http://127.0.0.1:51508/shutdown to shutdown server) 2012-05-22 17:13:48,346 PID: 1238 webtools.py:190 - INFO - Browsing http://127.0.0.1:51508/spothistory.html using 'default'... 2012-05-22 17:13:49,072 PID: 1238 webtools.py:56 - INFO - Shutting down server... 2012-05-22 17:14:22,059 PID: 1245 config.py:551 - DEBUG - Loading config 2012-05-22 17:14:22,059 PID: 1245 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 17:14:22,062 PID: 1245 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 17:14:23,963 PID: 1245 start.py:176 - INFO - Using default cluster template: sfmcluster 2012-05-22 17:14:23,963 PID: 1245 cluster.py:1539 - INFO - Validating cluster template settings... 2012-05-22 17:14:30,117 PID: 1245 cluster.py:1555 - INFO - Cluster template settings are valid 2012-05-22 17:14:30,118 PID: 1245 cluster.py:1427 - INFO - Starting cluster... 2012-05-22 17:14:30,118 PID: 1245 cluster.py:952 - INFO - Launching a 1-node cluster... 2012-05-22 17:14:30,118 PID: 1245 cluster.py:1004 - INFO - Launching master node (ami: ami-4583572c, type: cg1.4xlarge)... 2012-05-22 17:14:30,737 PID: 1245 awsutils.py:165 - INFO - Creating security group @sc-sfmpc... 2012-05-22 17:14:37,791 PID: 1245 awsutils.py:275 - INFO - Creating placement group @sc-sfmpc... 2012-05-22 17:14:40,007 PID: 1245 cluster.py:772 - INFO - SpotInstanceRequest:sir-fe09f811 2012-05-22 17:14:40,007 PID: 1245 utils.py:93 - INFO - Starting cluster took 0.165 mins 2012-05-22 17:14:45,796 PID: 1251 config.py:551 - DEBUG - Loading config 2012-05-22 17:14:45,796 PID: 1251 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 17:14:45,798 PID: 1251 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 17:14:49,183 PID: 1251 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 17:14:49,183 PID: 1251 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 17:14:57,044 PID: 1254 config.py:551 - DEBUG - Loading config 2012-05-22 17:14:57,044 PID: 1254 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 17:14:57,046 PID: 1254 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 17:15:00,534 PID: 1254 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 17:15:00,534 PID: 1254 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 17:15:50,346 PID: 1257 config.py:551 - DEBUG - Loading config 2012-05-22 17:15:50,347 PID: 1257 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 17:15:50,348 PID: 1257 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 17:15:55,241 PID: 1257 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 17:15:55,242 PID: 1257 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 17:43:00,510 PID: 1383 config.py:551 - DEBUG - Loading config 2012-05-22 17:43:00,510 PID: 1383 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 17:43:00,512 PID: 1383 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 17:43:15,117 PID: 1383 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 17:43:15,117 PID: 1383 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-22 17:43:23,799 PID: 1383 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 17:43:26,710 PID: 1383 node.py:910 - DEBUG - Using native OpenSSH client 2012-05-22 17:43:26,710 PID: 1383 node.py:919 - DEBUG - ssh_cmd: ssh -i /Users/mhelal/.ssh/mhelal.rsa root@ec2-50-19-38-201.compute-1.amazonaws.com 2012-05-22 17:43:51,393 PID: 1398 config.py:551 - DEBUG - Loading config 2012-05-22 17:43:51,394 PID: 1398 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 17:43:51,395 PID: 1398 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 17:44:00,905 PID: 1398 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 17:44:00,905 PID: 1398 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-22 17:44:00,906 PID: 1398 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 17:44:07,104 PID: 1398 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-22 17:44:07,106 PID: 1398 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-22 17:44:07,106 PID: 1398 __init__.py:97 - DEBUG - connecting to host ec2-50-19-38-201.compute-1.amazonaws.com on port 22 as user root 2012-05-22 17:44:22,310 PID: 1398 __init__.py:186 - DEBUG - creating sftp connection 2012-05-22 17:45:01,140 PID: 1407 config.py:551 - DEBUG - Loading config 2012-05-22 17:45:01,140 PID: 1407 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 17:45:01,142 PID: 1407 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 17:45:11,304 PID: 1407 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 17:45:11,305 PID: 1407 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-22 17:45:11,305 PID: 1407 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 17:45:14,943 PID: 1407 node.py:910 - DEBUG - Using native OpenSSH client 2012-05-22 17:45:14,943 PID: 1407 node.py:919 - DEBUG - ssh_cmd: ssh -i /Users/mhelal/.ssh/mhelal.rsa root@ec2-50-19-38-201.compute-1.amazonaws.com 2012-05-22 17:45:59,975 PID: 1413 config.py:551 - DEBUG - Loading config 2012-05-22 17:45:59,975 PID: 1413 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 17:45:59,977 PID: 1413 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 17:46:11,637 PID: 1413 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 17:46:11,638 PID: 1413 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-22 17:46:11,638 PID: 1413 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 17:46:16,580 PID: 1413 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-22 17:46:16,582 PID: 1413 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-22 17:46:16,582 PID: 1413 __init__.py:97 - DEBUG - connecting to host ec2-50-19-38-201.compute-1.amazonaws.com on port 22 as user root 2012-05-22 17:46:31,648 PID: 1413 __init__.py:186 - DEBUG - creating sftp connection 2012-05-22 17:55:26,307 PID: 1454 config.py:551 - DEBUG - Loading config 2012-05-22 17:55:26,307 PID: 1454 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 17:55:26,309 PID: 1454 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 17:55:34,440 PID: 1454 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 17:55:34,440 PID: 1454 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-22 17:55:34,440 PID: 1454 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 17:55:34,441 PID: 1454 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-22 17:55:34,442 PID: 1454 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-22 17:55:34,442 PID: 1454 __init__.py:194 - DEBUG - creating scp connection 2012-05-22 17:55:34,442 PID: 1454 __init__.py:97 - DEBUG - connecting to host ec2-50-19-38-201.compute-1.amazonaws.com on port 22 as user root 2012-05-22 17:55:40,512 PID: 1454 __init__.py:186 - DEBUG - creating sftp connection 2012-05-22 17:57:52,949 PID: 1469 config.py:551 - DEBUG - Loading config 2012-05-22 17:57:52,949 PID: 1469 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 17:57:52,951 PID: 1469 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 17:57:59,326 PID: 1469 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 17:57:59,326 PID: 1469 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-22 17:57:59,326 PID: 1469 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 17:57:59,327 PID: 1469 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-22 17:57:59,328 PID: 1469 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-22 17:57:59,328 PID: 1469 __init__.py:194 - DEBUG - creating scp connection 2012-05-22 17:57:59,328 PID: 1469 __init__.py:97 - DEBUG - connecting to host ec2-50-19-38-201.compute-1.amazonaws.com on port 22 as user root 2012-05-22 17:58:05,288 PID: 1469 __init__.py:186 - DEBUG - creating sftp connection 2012-05-22 18:00:40,751 PID: 1469 cli.py:279 - ERROR - Timeout waiting for scp response 2012-05-22 18:00:48,727 PID: 1483 config.py:551 - DEBUG - Loading config 2012-05-22 18:00:48,727 PID: 1483 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 18:00:48,729 PID: 1483 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 18:00:54,769 PID: 1483 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 18:00:54,769 PID: 1483 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-22 18:00:54,770 PID: 1483 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 18:00:54,770 PID: 1483 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-22 18:00:54,772 PID: 1483 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-22 18:00:54,772 PID: 1483 __init__.py:194 - DEBUG - creating scp connection 2012-05-22 18:00:54,772 PID: 1483 __init__.py:97 - DEBUG - connecting to host ec2-50-19-38-201.compute-1.amazonaws.com on port 22 as user root 2012-05-22 18:01:01,346 PID: 1483 __init__.py:186 - DEBUG - creating sftp connection 2012-05-22 18:05:34,175 PID: 1507 config.py:551 - DEBUG - Loading config 2012-05-22 18:05:34,175 PID: 1507 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 18:05:34,177 PID: 1507 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 18:05:41,353 PID: 1507 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 18:05:41,354 PID: 1507 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-22 18:05:41,354 PID: 1507 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 18:05:41,355 PID: 1507 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-22 18:05:41,356 PID: 1507 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-22 18:05:41,356 PID: 1507 __init__.py:194 - DEBUG - creating scp connection 2012-05-22 18:05:41,356 PID: 1507 __init__.py:97 - DEBUG - connecting to host ec2-50-19-38-201.compute-1.amazonaws.com on port 22 as user root 2012-05-22 18:05:48,336 PID: 1507 __init__.py:186 - DEBUG - creating sftp connection 2012-05-22 18:15:22,032 PID: 1564 config.py:551 - DEBUG - Loading config 2012-05-22 18:15:22,032 PID: 1564 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 18:15:22,034 PID: 1564 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 18:15:28,696 PID: 1564 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 18:15:28,696 PID: 1564 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-22 18:15:28,697 PID: 1564 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 18:15:28,896 PID: 1564 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-22 18:15:28,898 PID: 1564 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-22 18:15:28,898 PID: 1564 __init__.py:194 - DEBUG - creating scp connection 2012-05-22 18:15:28,898 PID: 1564 __init__.py:97 - DEBUG - connecting to host ec2-50-19-38-201.compute-1.amazonaws.com on port 22 as user root 2012-05-22 18:15:35,451 PID: 1564 __init__.py:186 - DEBUG - creating sftp connection 2012-05-22 18:37:09,607 PID: 1686 config.py:551 - DEBUG - Loading config 2012-05-22 18:37:09,607 PID: 1686 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 18:37:09,609 PID: 1686 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 18:37:17,667 PID: 1686 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 18:37:17,668 PID: 1686 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-22 18:37:17,668 PID: 1686 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 18:37:17,668 PID: 1686 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-22 18:37:17,670 PID: 1686 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-22 18:37:17,670 PID: 1686 __init__.py:194 - DEBUG - creating scp connection 2012-05-22 18:37:17,670 PID: 1686 __init__.py:97 - DEBUG - connecting to host ec2-50-19-38-201.compute-1.amazonaws.com on port 22 as user root 2012-05-22 18:37:24,538 PID: 1686 __init__.py:186 - DEBUG - creating sftp connection 2012-05-22 18:38:05,925 PID: 1686 cli.py:279 - ERROR - Timeout waiting for scp response 2012-05-22 18:38:09,708 PID: 1692 config.py:551 - DEBUG - Loading config 2012-05-22 18:38:09,708 PID: 1692 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 18:38:09,710 PID: 1692 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 18:38:15,417 PID: 1692 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 18:38:15,418 PID: 1692 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-22 18:38:15,418 PID: 1692 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 18:38:15,419 PID: 1692 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-22 18:38:15,420 PID: 1692 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-22 18:38:15,420 PID: 1692 __init__.py:194 - DEBUG - creating scp connection 2012-05-22 18:38:15,420 PID: 1692 __init__.py:97 - DEBUG - connecting to host ec2-50-19-38-201.compute-1.amazonaws.com on port 22 as user root 2012-05-22 18:38:21,819 PID: 1692 __init__.py:186 - DEBUG - creating sftp connection 2012-05-22 18:39:02,288 PID: 1700 config.py:551 - DEBUG - Loading config 2012-05-22 18:39:02,288 PID: 1700 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 18:39:02,290 PID: 1700 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 18:39:09,147 PID: 1700 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 18:39:09,148 PID: 1700 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-22 18:39:09,148 PID: 1700 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 18:39:09,148 PID: 1700 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-22 18:39:09,149 PID: 1700 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-22 18:39:09,150 PID: 1700 __init__.py:194 - DEBUG - creating scp connection 2012-05-22 18:39:09,150 PID: 1700 __init__.py:97 - DEBUG - connecting to host ec2-50-19-38-201.compute-1.amazonaws.com on port 22 as user root 2012-05-22 18:39:15,546 PID: 1700 __init__.py:186 - DEBUG - creating sftp connection 2012-05-22 18:39:50,566 PID: 1704 config.py:551 - DEBUG - Loading config 2012-05-22 18:39:50,566 PID: 1704 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 18:39:50,568 PID: 1704 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 18:39:55,580 PID: 1704 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 18:39:55,580 PID: 1704 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-22 18:39:55,580 PID: 1704 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 18:39:55,581 PID: 1704 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-22 18:39:55,582 PID: 1704 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-22 18:39:55,582 PID: 1704 __init__.py:194 - DEBUG - creating scp connection 2012-05-22 18:39:55,583 PID: 1704 __init__.py:97 - DEBUG - connecting to host ec2-50-19-38-201.compute-1.amazonaws.com on port 22 as user root 2012-05-22 18:40:01,670 PID: 1704 __init__.py:186 - DEBUG - creating sftp connection 2012-05-22 18:48:06,195 PID: 1744 config.py:551 - DEBUG - Loading config 2012-05-22 18:48:06,196 PID: 1744 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 18:48:06,198 PID: 1744 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 18:48:14,252 PID: 1744 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 18:48:14,253 PID: 1744 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-22 18:48:14,253 PID: 1744 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 18:48:14,254 PID: 1744 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-22 18:48:14,255 PID: 1744 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-22 18:48:14,255 PID: 1744 __init__.py:194 - DEBUG - creating scp connection 2012-05-22 18:48:14,255 PID: 1744 __init__.py:97 - DEBUG - connecting to host ec2-50-19-38-201.compute-1.amazonaws.com on port 22 as user root 2012-05-22 18:48:20,776 PID: 1744 __init__.py:186 - DEBUG - creating sftp connection 2012-05-22 18:48:33,909 PID: 1747 config.py:551 - DEBUG - Loading config 2012-05-22 18:48:33,910 PID: 1747 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 18:48:33,912 PID: 1747 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 18:48:39,122 PID: 1747 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 18:48:39,122 PID: 1747 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-22 18:48:39,123 PID: 1747 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 18:48:39,123 PID: 1747 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-22 18:48:39,124 PID: 1747 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-22 18:48:39,124 PID: 1747 __init__.py:194 - DEBUG - creating scp connection 2012-05-22 18:48:39,125 PID: 1747 __init__.py:97 - DEBUG - connecting to host ec2-50-19-38-201.compute-1.amazonaws.com on port 22 as user root 2012-05-22 18:48:45,314 PID: 1747 __init__.py:186 - DEBUG - creating sftp connection 2012-05-22 18:48:51,829 PID: 1750 config.py:551 - DEBUG - Loading config 2012-05-22 18:48:51,830 PID: 1750 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 18:48:51,831 PID: 1750 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 18:48:58,316 PID: 1750 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 18:48:58,317 PID: 1750 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-22 18:48:58,317 PID: 1750 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 18:48:58,317 PID: 1750 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-22 18:48:58,319 PID: 1750 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-22 18:48:58,319 PID: 1750 __init__.py:194 - DEBUG - creating scp connection 2012-05-22 18:48:58,319 PID: 1750 __init__.py:97 - DEBUG - connecting to host ec2-50-19-38-201.compute-1.amazonaws.com on port 22 as user root 2012-05-22 18:49:04,585 PID: 1750 __init__.py:186 - DEBUG - creating sftp connection 2012-05-22 18:49:16,538 PID: 1753 config.py:551 - DEBUG - Loading config 2012-05-22 18:49:16,539 PID: 1753 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 18:49:16,541 PID: 1753 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 18:49:22,316 PID: 1753 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 18:49:22,317 PID: 1753 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-22 18:49:22,317 PID: 1753 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 18:49:22,317 PID: 1753 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-22 18:49:22,318 PID: 1753 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-22 18:49:22,319 PID: 1753 __init__.py:194 - DEBUG - creating scp connection 2012-05-22 18:49:22,319 PID: 1753 __init__.py:97 - DEBUG - connecting to host ec2-50-19-38-201.compute-1.amazonaws.com on port 22 as user root 2012-05-22 18:49:28,288 PID: 1753 __init__.py:186 - DEBUG - creating sftp connection 2012-05-22 19:06:13,319 PID: 1881 config.py:551 - DEBUG - Loading config 2012-05-22 19:06:13,319 PID: 1881 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 19:06:13,321 PID: 1881 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 19:06:20,811 PID: 1881 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 19:06:20,811 PID: 1881 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-22 19:06:20,811 PID: 1881 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 19:06:20,812 PID: 1881 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-22 19:06:20,812 PID: 1881 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-22 19:06:20,813 PID: 1881 __init__.py:194 - DEBUG - creating scp connection 2012-05-22 19:06:20,813 PID: 1881 __init__.py:97 - DEBUG - connecting to host ec2-50-19-38-201.compute-1.amazonaws.com on port 22 as user root 2012-05-22 19:06:26,287 PID: 1881 __init__.py:186 - DEBUG - creating sftp connection 2012-05-22 19:42:16,996 PID: 2146 config.py:551 - DEBUG - Loading config 2012-05-22 19:42:16,996 PID: 2146 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 19:42:16,999 PID: 2146 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 19:42:22,783 PID: 2146 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 19:42:22,784 PID: 2146 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-22 19:42:22,784 PID: 2146 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 19:42:22,784 PID: 2146 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-22 19:42:22,786 PID: 2146 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-22 19:42:22,786 PID: 2146 __init__.py:194 - DEBUG - creating scp connection 2012-05-22 19:42:22,786 PID: 2146 __init__.py:97 - DEBUG - connecting to host ec2-50-19-38-201.compute-1.amazonaws.com on port 22 as user root 2012-05-22 19:42:31,235 PID: 2146 __init__.py:186 - DEBUG - creating sftp connection 2012-05-22 19:42:58,551 PID: 2151 config.py:551 - DEBUG - Loading config 2012-05-22 19:42:58,551 PID: 2151 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 19:42:58,554 PID: 2151 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 19:43:04,546 PID: 2151 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 19:43:04,547 PID: 2151 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-22 19:43:04,547 PID: 2151 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 19:43:04,547 PID: 2151 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-22 19:43:04,549 PID: 2151 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-22 19:43:04,549 PID: 2151 __init__.py:194 - DEBUG - creating scp connection 2012-05-22 19:43:04,549 PID: 2151 __init__.py:97 - DEBUG - connecting to host ec2-50-19-38-201.compute-1.amazonaws.com on port 22 as user root 2012-05-22 19:43:13,108 PID: 2151 __init__.py:186 - DEBUG - creating sftp connection 2012-05-22 19:43:18,733 PID: 2154 config.py:551 - DEBUG - Loading config 2012-05-22 19:43:18,734 PID: 2154 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 19:43:18,737 PID: 2154 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 19:43:23,883 PID: 2154 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 19:43:23,884 PID: 2154 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-22 19:43:23,884 PID: 2154 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 19:43:23,885 PID: 2154 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-22 19:43:23,886 PID: 2154 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-22 19:43:23,886 PID: 2154 __init__.py:194 - DEBUG - creating scp connection 2012-05-22 19:43:23,886 PID: 2154 __init__.py:97 - DEBUG - connecting to host ec2-50-19-38-201.compute-1.amazonaws.com on port 22 as user root 2012-05-22 19:43:32,448 PID: 2154 __init__.py:186 - DEBUG - creating sftp connection 2012-05-22 19:43:37,644 PID: 2157 config.py:551 - DEBUG - Loading config 2012-05-22 19:43:37,644 PID: 2157 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 19:43:37,648 PID: 2157 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 19:43:45,238 PID: 2157 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 19:43:45,238 PID: 2157 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-22 19:43:45,239 PID: 2157 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 19:43:45,239 PID: 2157 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-22 19:43:45,240 PID: 2157 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-22 19:43:45,241 PID: 2157 __init__.py:194 - DEBUG - creating scp connection 2012-05-22 19:43:45,241 PID: 2157 __init__.py:97 - DEBUG - connecting to host ec2-50-19-38-201.compute-1.amazonaws.com on port 22 as user root 2012-05-22 19:43:53,982 PID: 2157 __init__.py:186 - DEBUG - creating sftp connection 2012-05-22 20:19:33,353 PID: 2313 config.py:551 - DEBUG - Loading config 2012-05-22 20:19:33,353 PID: 2313 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 20:19:33,355 PID: 2313 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 20:19:40,206 PID: 2313 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 20:19:40,207 PID: 2313 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-22 20:19:40,207 PID: 2313 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 20:19:40,207 PID: 2313 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-22 20:19:40,208 PID: 2313 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-22 20:19:40,208 PID: 2313 __init__.py:194 - DEBUG - creating scp connection 2012-05-22 20:19:40,208 PID: 2313 __init__.py:97 - DEBUG - connecting to host ec2-50-19-38-201.compute-1.amazonaws.com on port 22 as user root 2012-05-22 20:19:47,865 PID: 2313 __init__.py:186 - DEBUG - creating sftp connection 2012-05-22 20:20:01,789 PID: 2318 config.py:551 - DEBUG - Loading config 2012-05-22 20:20:01,789 PID: 2318 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 20:20:01,792 PID: 2318 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 20:20:07,211 PID: 2318 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 20:20:07,211 PID: 2318 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-22 20:20:07,212 PID: 2318 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 20:20:07,212 PID: 2318 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-22 20:20:07,213 PID: 2318 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-22 20:20:07,214 PID: 2318 __init__.py:194 - DEBUG - creating scp connection 2012-05-22 20:20:07,214 PID: 2318 __init__.py:97 - DEBUG - connecting to host ec2-50-19-38-201.compute-1.amazonaws.com on port 22 as user root 2012-05-22 20:20:13,028 PID: 2318 __init__.py:186 - DEBUG - creating sftp connection 2012-05-22 20:49:10,184 PID: 2455 config.py:551 - DEBUG - Loading config 2012-05-22 20:49:10,185 PID: 2455 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 20:49:10,186 PID: 2455 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 20:49:14,663 PID: 2455 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 20:49:14,664 PID: 2455 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-22 20:49:14,664 PID: 2455 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 20:49:16,169 PID: 2455 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-22 20:49:16,170 PID: 2455 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-22 20:49:16,170 PID: 2455 __init__.py:97 - DEBUG - connecting to host ec2-50-19-38-201.compute-1.amazonaws.com on port 22 as user root 2012-05-22 20:49:22,153 PID: 2455 __init__.py:186 - DEBUG - creating sftp connection 2012-05-22 20:49:44,909 PID: 2460 config.py:551 - DEBUG - Loading config 2012-05-22 20:49:44,910 PID: 2460 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 20:49:44,912 PID: 2460 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 20:49:48,139 PID: 2460 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 20:49:48,139 PID: 2460 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-22 20:49:48,140 PID: 2460 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 20:49:49,683 PID: 2460 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-22 20:49:49,684 PID: 2460 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-22 20:49:49,684 PID: 2460 __init__.py:97 - DEBUG - connecting to host ec2-50-19-38-201.compute-1.amazonaws.com on port 22 as user root 2012-05-22 20:49:56,129 PID: 2460 __init__.py:186 - DEBUG - creating sftp connection 2012-05-22 20:50:37,845 PID: 2468 config.py:551 - DEBUG - Loading config 2012-05-22 20:50:37,845 PID: 2468 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 20:50:37,847 PID: 2468 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 20:50:42,520 PID: 2468 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 20:50:42,520 PID: 2468 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-22 20:50:42,521 PID: 2468 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 20:50:43,462 PID: 2468 cluster.py:664 - DEBUG - existing nodes: {u'i-81d384e7': } 2012-05-22 20:50:43,462 PID: 2468 cluster.py:667 - DEBUG - updating existing node i-81d384e7 in self._nodes 2012-05-22 20:50:43,462 PID: 2468 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 20:50:44,371 PID: 2468 cluster.py:664 - DEBUG - existing nodes: {u'i-81d384e7': } 2012-05-22 20:50:44,371 PID: 2468 cluster.py:667 - DEBUG - updating existing node i-81d384e7 in self._nodes 2012-05-22 20:50:44,371 PID: 2468 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 20:50:44,372 PID: 2468 cli.py:279 - ERROR - Cluster 'sfmcluster' does not contain any 'stoppable' nodes and can only be terminated. Please use the 'terminate' command instead to destroy the cluster. Pass --help for more info 2012-05-22 21:10:42,667 PID: 2585 config.py:551 - DEBUG - Loading config 2012-05-22 21:10:42,667 PID: 2585 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 21:10:42,669 PID: 2585 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 21:10:48,598 PID: 2585 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 21:10:48,599 PID: 2585 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-22 21:10:48,599 PID: 2585 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 21:10:49,506 PID: 2585 node.py:910 - DEBUG - Using native OpenSSH client 2012-05-22 21:10:49,506 PID: 2585 node.py:919 - DEBUG - ssh_cmd: ssh -i /Users/mhelal/.ssh/mhelal.rsa root@ec2-50-19-38-201.compute-1.amazonaws.com 2012-05-22 21:20:43,092 PID: 2652 config.py:551 - DEBUG - Loading config 2012-05-22 21:20:43,092 PID: 2652 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 21:20:43,094 PID: 2652 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 21:20:49,571 PID: 2652 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 21:20:49,571 PID: 2652 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-22 21:20:49,571 PID: 2652 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 21:20:50,472 PID: 2652 node.py:910 - DEBUG - Using native OpenSSH client 2012-05-22 21:20:50,472 PID: 2652 node.py:919 - DEBUG - ssh_cmd: ssh -i /Users/mhelal/.ssh/mhelal.rsa root@ec2-50-19-38-201.compute-1.amazonaws.com 2012-05-22 22:04:06,297 PID: 2887 config.py:551 - DEBUG - Loading config 2012-05-22 22:04:06,297 PID: 2887 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 22:04:06,299 PID: 2887 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 22:04:18,356 PID: 2887 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 22:04:18,356 PID: 2887 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-22 22:04:18,357 PID: 2887 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 22:04:19,154 PID: 2887 node.py:910 - DEBUG - Using native OpenSSH client 2012-05-22 22:04:19,155 PID: 2887 node.py:919 - DEBUG - ssh_cmd: ssh -i /Users/mhelal/.ssh/mhelal.rsa root@ec2-50-19-38-201.compute-1.amazonaws.com 2012-05-22 22:36:01,874 PID: 3227 config.py:551 - DEBUG - Loading config 2012-05-22 22:36:01,874 PID: 3227 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 22:36:25,000 PID: 3229 config.py:551 - DEBUG - Loading config 2012-05-22 22:36:25,000 PID: 3229 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 22:36:38,930 PID: 3231 config.py:551 - DEBUG - Loading config 2012-05-22 22:36:38,931 PID: 3231 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 22:36:38,933 PID: 3231 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 22:36:46,754 PID: 3231 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 22:36:46,754 PID: 3231 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-22 22:36:46,754 PID: 3231 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 22:36:46,754 PID: 3231 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-22 22:36:46,755 PID: 3231 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-22 22:36:46,755 PID: 3231 __init__.py:186 - DEBUG - creating sftp connection 2012-05-22 22:36:46,756 PID: 3231 __init__.py:97 - DEBUG - connecting to host ec2-50-19-38-201.compute-1.amazonaws.com on port 22 as user root 2012-05-22 22:36:53,751 PID: 3231 __init__.py:186 - DEBUG - creating sftp connection 2012-05-22 22:36:58,341 PID: 3231 __init__.py:194 - DEBUG - creating scp connection 2012-05-22 23:14:06,210 PID: 3413 config.py:551 - DEBUG - Loading config 2012-05-22 23:14:06,210 PID: 3413 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 23:14:06,212 PID: 3413 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 23:14:11,592 PID: 3413 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 23:14:11,592 PID: 3413 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-22 23:14:11,593 PID: 3413 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 23:14:12,151 PID: 3413 node.py:910 - DEBUG - Using native OpenSSH client 2012-05-22 23:14:12,151 PID: 3413 node.py:919 - DEBUG - ssh_cmd: ssh -i /Users/mhelal/.ssh/mhelal.rsa root@ec2-50-19-38-201.compute-1.amazonaws.com 2012-05-22 23:34:14,973 PID: 3515 config.py:551 - DEBUG - Loading config 2012-05-22 23:34:14,974 PID: 3515 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 23:34:14,976 PID: 3515 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 23:34:20,057 PID: 3515 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 23:34:20,058 PID: 3515 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-22 23:34:20,058 PID: 3515 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 23:34:20,058 PID: 3515 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-22 23:34:20,060 PID: 3515 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-22 23:34:20,060 PID: 3515 __init__.py:186 - DEBUG - creating sftp connection 2012-05-22 23:34:20,060 PID: 3515 __init__.py:97 - DEBUG - connecting to host ec2-50-19-38-201.compute-1.amazonaws.com on port 22 as user root 2012-05-22 23:34:26,762 PID: 3515 __init__.py:186 - DEBUG - creating sftp connection 2012-05-22 23:34:30,856 PID: 3515 __init__.py:194 - DEBUG - creating scp connection 2012-05-22 23:35:25,157 PID: 3522 config.py:551 - DEBUG - Loading config 2012-05-22 23:35:25,158 PID: 3522 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 23:35:25,160 PID: 3522 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 23:35:29,642 PID: 3522 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 23:35:29,642 PID: 3522 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-22 23:35:29,643 PID: 3522 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 23:35:29,643 PID: 3522 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-22 23:35:29,644 PID: 3522 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-22 23:35:29,644 PID: 3522 __init__.py:186 - DEBUG - creating sftp connection 2012-05-22 23:35:29,645 PID: 3522 __init__.py:97 - DEBUG - connecting to host ec2-50-19-38-201.compute-1.amazonaws.com on port 22 as user root 2012-05-22 23:35:35,780 PID: 3522 __init__.py:186 - DEBUG - creating sftp connection 2012-05-22 23:35:38,850 PID: 3522 __init__.py:194 - DEBUG - creating scp connection 2012-05-22 23:37:58,505 PID: 3535 config.py:551 - DEBUG - Loading config 2012-05-22 23:37:58,506 PID: 3535 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 23:37:58,508 PID: 3535 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 23:38:03,010 PID: 3535 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 23:38:03,010 PID: 3535 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-22 23:38:03,011 PID: 3535 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 23:38:03,011 PID: 3535 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-22 23:38:03,012 PID: 3535 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-22 23:38:03,012 PID: 3535 __init__.py:186 - DEBUG - creating sftp connection 2012-05-22 23:38:03,012 PID: 3535 __init__.py:97 - DEBUG - connecting to host ec2-50-19-38-201.compute-1.amazonaws.com on port 22 as user root 2012-05-22 23:38:09,503 PID: 3535 __init__.py:186 - DEBUG - creating sftp connection 2012-05-22 23:38:12,457 PID: 3535 __init__.py:194 - DEBUG - creating scp connection 2012-05-22 23:57:00,013 PID: 3706 config.py:551 - DEBUG - Loading config 2012-05-22 23:57:00,013 PID: 3706 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-22 23:57:00,015 PID: 3706 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-22 23:57:07,450 PID: 3706 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-22 23:57:07,451 PID: 3706 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-22 23:57:07,451 PID: 3706 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-22 23:57:07,452 PID: 3706 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-22 23:57:07,453 PID: 3706 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-22 23:57:07,453 PID: 3706 __init__.py:186 - DEBUG - creating sftp connection 2012-05-22 23:57:07,453 PID: 3706 __init__.py:97 - DEBUG - connecting to host ec2-50-19-38-201.compute-1.amazonaws.com on port 22 as user root 2012-05-22 23:57:13,897 PID: 3706 __init__.py:186 - DEBUG - creating sftp connection 2012-05-22 23:57:16,969 PID: 3706 __init__.py:194 - DEBUG - creating scp connection 2012-05-23 00:19:41,950 PID: 3820 config.py:551 - DEBUG - Loading config 2012-05-23 00:19:41,950 PID: 3820 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-23 00:19:41,952 PID: 3820 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-23 00:19:47,017 PID: 3820 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-23 00:19:47,017 PID: 3820 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-23 00:19:47,017 PID: 3820 cluster.py:672 - DEBUG - adding node i-8b4212ed to self._nodes list 2012-05-23 00:19:48,036 PID: 3820 cli.py:287 - DEBUG - Traceback (most recent call last): File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/cli.py", line 255, in main sc.execute(args) File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/commands/listclusters.py", line 19, in execute show_ssh_status=self.opts.show_ssh_status) File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/cluster.py", line 247, in list_clusters nodes = cl.nodes File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/cluster.py", line 674, in nodes if n.is_master(): File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/node.py", line 764, in is_master return self.alias == "master" File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/node.py", line 108, in alias user_data = self._get_user_data(tries=5) File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/node.py", line 88, in _get_user_data user_data = self.ec2.get_instance_user_data(self.id) File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/awsutils.py", line 450, in get_instance_user_data return base64.b64decode(user_data) File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/base64.py", line 73, in b64decode return binascii.a2b_base64(s) TypeError: a2b_base64() argument 1 must be string or read-only character buffer, not None 2012-05-23 00:19:48,222 PID: 3820 cli.py:165 - ERROR - Oops! Looks like you've found a bug in StarCluster 2012-05-23 00:19:48,222 PID: 3820 cli.py:166 - ERROR - Crash report written to: /Users/mhelal/.starcluster/logs/crash-report-3820.txt 2012-05-23 00:19:48,222 PID: 3820 cli.py:167 - ERROR - Please remove any sensitive data from the crash report 2012-05-23 00:19:48,223 PID: 3820 cli.py:168 - ERROR - and submit it to starcluster@mit.edu 2012-05-23 00:28:06,361 PID: 3882 config.py:551 - DEBUG - Loading config 2012-05-23 00:28:06,361 PID: 3882 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-23 00:28:06,363 PID: 3882 cli.py:79 - ERROR - missing required option keyname in section "cluster mysfmcluster" 2012-05-23 00:36:11,522 PID: 3922 config.py:551 - DEBUG - Loading config 2012-05-23 00:36:11,522 PID: 3922 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-23 00:36:11,524 PID: 3922 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-23 00:36:14,942 PID: 3922 start.py:176 - INFO - Using default cluster template: sfmcluster 2012-05-23 00:36:14,943 PID: 3922 cluster.py:1539 - INFO - Validating cluster template settings... 2012-05-23 00:36:21,407 PID: 3922 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-23 00:36:21,408 PID: 3922 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-23 00:36:21,408 PID: 3922 cli.py:279 - ERROR - Cluster settings are not valid: volume vol-69bd4807 is not available (status: in-use) 2012-05-23 00:37:29,353 PID: 3934 config.py:551 - DEBUG - Loading config 2012-05-23 00:37:29,353 PID: 3934 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-23 00:37:29,355 PID: 3934 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-23 00:38:04,776 PID: 3938 config.py:551 - DEBUG - Loading config 2012-05-23 00:38:04,777 PID: 3938 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-23 00:38:04,779 PID: 3938 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-23 00:38:35,239 PID: 3941 config.py:551 - DEBUG - Loading config 2012-05-23 00:38:35,240 PID: 3941 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-23 00:38:35,242 PID: 3941 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-23 00:38:52,614 PID: 3944 config.py:551 - DEBUG - Loading config 2012-05-23 00:38:52,614 PID: 3944 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-23 00:38:52,616 PID: 3944 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-23 00:40:27,520 PID: 3954 config.py:551 - DEBUG - Loading config 2012-05-23 00:40:27,520 PID: 3954 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-23 00:40:27,522 PID: 3954 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-23 00:40:53,937 PID: 3957 config.py:551 - DEBUG - Loading config 2012-05-23 00:40:53,938 PID: 3957 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-23 00:40:53,940 PID: 3957 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-23 00:40:59,528 PID: 3957 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-23 00:40:59,528 PID: 3957 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-23 00:40:59,528 PID: 3957 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-23 00:41:00,760 PID: 3957 node.py:910 - DEBUG - Using native OpenSSH client 2012-05-23 00:41:00,760 PID: 3957 node.py:919 - DEBUG - ssh_cmd: ssh -i /Users/mhelal/.ssh/mhelal.rsa root@ec2-50-19-38-201.compute-1.amazonaws.com 2012-05-23 00:45:45,104 PID: 4002 config.py:551 - DEBUG - Loading config 2012-05-23 00:45:45,104 PID: 4002 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-23 00:45:45,106 PID: 4002 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-23 00:45:50,138 PID: 4002 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-23 00:45:50,138 PID: 4002 cluster.py:672 - DEBUG - adding node i-81d384e7 to self._nodes list 2012-05-23 00:45:50,139 PID: 4002 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-23 00:51:10,138 PID: 4002 cluster.py:664 - DEBUG - existing nodes: {u'i-81d384e7': } 2012-05-23 00:51:10,138 PID: 4002 cluster.py:667 - DEBUG - updating existing node i-81d384e7 in self._nodes 2012-05-23 00:51:10,138 PID: 4002 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-23 00:51:11,264 PID: 4002 cluster.py:664 - DEBUG - existing nodes: {u'i-81d384e7': } 2012-05-23 00:51:11,264 PID: 4002 cluster.py:667 - DEBUG - updating existing node i-81d384e7 in self._nodes 2012-05-23 00:51:11,264 PID: 4002 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-23 00:51:11,265 PID: 4002 node.py:828 - INFO - Terminating node: master (i-81d384e7) 2012-05-23 00:51:13,517 PID: 4002 cluster.py:1368 - INFO - Canceling spot instance request: sir-fe09f811 2012-05-23 00:51:15,719 PID: 4002 cluster.py:1140 - INFO - Waiting for cluster to terminate... 2012-05-23 00:54:25,763 PID: 4002 cluster.py:1377 - INFO - Removing @sc-sfmpc placement group 2012-05-23 00:54:26,586 PID: 4002 cluster.py:1380 - INFO - Removing @sc-sfmpc security group 2012-05-23 00:57:28,672 PID: 4076 config.py:551 - DEBUG - Loading config 2012-05-23 00:57:28,672 PID: 4076 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-23 00:57:28,675 PID: 4076 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-23 00:57:31,165 PID: 4076 start.py:176 - INFO - Using default cluster template: sfmcluster 2012-05-23 00:57:31,166 PID: 4076 cluster.py:1539 - INFO - Validating cluster template settings... 2012-05-23 00:57:36,180 PID: 4076 cluster.py:1555 - INFO - Cluster template settings are valid 2012-05-23 00:57:36,180 PID: 4076 cluster.py:1427 - INFO - Starting cluster... 2012-05-23 00:57:36,180 PID: 4076 cluster.py:952 - INFO - Launching a 1-node cluster... 2012-05-23 00:57:36,181 PID: 4076 cluster.py:1004 - INFO - Launching master node (ami: ami-4583572c, type: cg1.4xlarge)... 2012-05-23 00:57:36,775 PID: 4076 awsutils.py:165 - INFO - Creating security group @sc-mysfmcluster... 2012-05-23 00:57:42,293 PID: 4076 awsutils.py:275 - INFO - Creating placement group @sc-mysfmcluster... 2012-05-23 00:57:44,260 PID: 4076 cluster.py:772 - INFO - SpotInstanceRequest:sir-0a5e0211 2012-05-23 00:57:44,260 PID: 4076 utils.py:93 - INFO - Starting cluster took 0.135 mins 2012-05-23 00:59:20,404 PID: 4088 config.py:551 - DEBUG - Loading config 2012-05-23 00:59:20,405 PID: 4088 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-23 00:59:20,407 PID: 4088 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-23 00:59:25,852 PID: 4088 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-23 00:59:25,853 PID: 4088 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-23 01:01:51,894 PID: 4088 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-23 01:01:51,894 PID: 4088 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-23 01:01:52,488 PID: 4088 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-23 01:01:52,489 PID: 4088 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-23 01:01:54,944 PID: 4088 cluster.py:1140 - INFO - Waiting for cluster to terminate... 2012-05-23 01:01:56,058 PID: 4088 cluster.py:1377 - INFO - Removing @sc-mysfmcluster placement group 2012-05-23 01:01:56,776 PID: 4088 cluster.py:1380 - INFO - Removing @sc-mysfmcluster security group 2012-05-23 01:02:11,980 PID: 4106 config.py:551 - DEBUG - Loading config 2012-05-23 01:02:11,980 PID: 4106 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-23 01:02:11,983 PID: 4106 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-23 01:02:13,330 PID: 4106 cli.py:279 - ERROR - cluster template sfmcluster does not exist 2012-05-23 01:02:50,516 PID: 4114 config.py:551 - DEBUG - Loading config 2012-05-23 01:02:50,516 PID: 4114 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-23 01:02:50,518 PID: 4114 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-23 01:02:51,804 PID: 4114 start.py:176 - INFO - Using default cluster template: mysfmcluster 2012-05-23 01:02:51,805 PID: 4114 cluster.py:1539 - INFO - Validating cluster template settings... 2012-05-23 01:02:58,699 PID: 4114 cluster.py:1555 - INFO - Cluster template settings are valid 2012-05-23 01:02:58,699 PID: 4114 cluster.py:1427 - INFO - Starting cluster... 2012-05-23 01:02:58,700 PID: 4114 cluster.py:952 - INFO - Launching a 1-node cluster... 2012-05-23 01:02:58,700 PID: 4114 cluster.py:1004 - INFO - Launching master node (ami: ami-fae74193, type: cg1.4xlarge)... 2012-05-23 01:02:59,383 PID: 4114 awsutils.py:165 - INFO - Creating security group @sc-mysfmcluster... 2012-05-23 01:03:05,296 PID: 4114 awsutils.py:275 - INFO - Creating placement group @sc-mysfmcluster... 2012-05-23 01:03:07,532 PID: 4114 cluster.py:772 - INFO - SpotInstanceRequest:sir-158bb214 2012-05-23 01:03:07,532 PID: 4114 utils.py:93 - INFO - Starting cluster took 0.147 mins 2012-05-23 01:03:16,451 PID: 4119 config.py:551 - DEBUG - Loading config 2012-05-23 01:03:16,452 PID: 4119 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-23 01:03:16,453 PID: 4119 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-23 01:03:20,344 PID: 4119 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-23 01:03:20,344 PID: 4119 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-23 01:05:07,783 PID: 4133 config.py:551 - DEBUG - Loading config 2012-05-23 01:05:07,783 PID: 4133 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-23 01:05:07,785 PID: 4133 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-23 01:05:11,394 PID: 4133 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-23 01:05:11,394 PID: 4133 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-23 01:05:16,866 PID: 4136 config.py:551 - DEBUG - Loading config 2012-05-23 01:05:16,866 PID: 4136 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-23 01:05:16,868 PID: 4136 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-23 01:05:20,155 PID: 4136 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-23 01:05:20,155 PID: 4136 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-23 01:05:51,688 PID: 4138 config.py:551 - DEBUG - Loading config 2012-05-23 01:05:51,688 PID: 4138 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-23 01:05:51,690 PID: 4138 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-23 01:05:55,178 PID: 4138 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-23 01:05:55,178 PID: 4138 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-23 01:07:46,449 PID: 4151 config.py:551 - DEBUG - Loading config 2012-05-23 01:07:46,450 PID: 4151 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-23 01:07:46,451 PID: 4151 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-23 01:07:51,468 PID: 4151 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-23 01:07:51,468 PID: 4151 cluster.py:672 - DEBUG - adding node i-91aefef7 to self._nodes list 2012-05-23 01:07:54,163 PID: 4151 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-23 01:07:56,067 PID: 4151 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-23 01:07:56,068 PID: 4151 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-23 01:07:56,068 PID: 4151 __init__.py:97 - DEBUG - connecting to host ec2-50-16-174-67.compute-1.amazonaws.com on port 22 as user root 2012-05-23 01:08:33,662 PID: 4155 config.py:551 - DEBUG - Loading config 2012-05-23 01:08:33,662 PID: 4155 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-23 01:08:33,664 PID: 4155 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-23 01:08:35,371 PID: 4155 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-23 01:08:39,359 PID: 4155 cli.py:272 - ERROR - cluster 'sfmcluster' does not exist 2012-05-23 01:08:39,360 PID: 4155 cli.py:276 - ERROR - (active clusters: mysfmcluster) 2012-05-23 01:08:43,928 PID: 4159 config.py:551 - DEBUG - Loading config 2012-05-23 01:08:43,929 PID: 4159 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-23 01:08:43,931 PID: 4159 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-23 01:08:50,208 PID: 4159 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-23 01:08:50,209 PID: 4159 cluster.py:672 - DEBUG - adding node i-91aefef7 to self._nodes list 2012-05-23 01:08:50,209 PID: 4159 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-23 01:08:50,897 PID: 4159 node.py:910 - DEBUG - Using native OpenSSH client 2012-05-23 01:08:50,898 PID: 4159 node.py:919 - DEBUG - ssh_cmd: ssh -i /Users/mhelal/.ssh/mhelal.rsa root@ec2-50-16-174-67.compute-1.amazonaws.com 2012-05-23 01:09:01,463 PID: 4163 config.py:551 - DEBUG - Loading config 2012-05-23 01:09:01,463 PID: 4163 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-23 01:09:01,465 PID: 4163 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-23 01:09:06,711 PID: 4163 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-23 01:09:06,711 PID: 4163 cluster.py:672 - DEBUG - adding node i-91aefef7 to self._nodes list 2012-05-23 01:09:06,711 PID: 4163 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-23 01:09:07,631 PID: 4163 node.py:910 - DEBUG - Using native OpenSSH client 2012-05-23 01:09:07,632 PID: 4163 node.py:919 - DEBUG - ssh_cmd: ssh -i /Users/mhelal/.ssh/mhelal.rsa root@ec2-50-16-174-67.compute-1.amazonaws.com 2012-05-23 02:33:18,405 PID: 4549 config.py:551 - DEBUG - Loading config 2012-05-23 02:33:18,405 PID: 4549 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-23 02:33:18,407 PID: 4549 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-23 02:33:23,563 PID: 4549 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-23 02:33:23,563 PID: 4549 cluster.py:672 - DEBUG - adding node i-91aefef7 to self._nodes list 2012-05-23 02:33:23,564 PID: 4549 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-23 02:33:36,158 PID: 4549 cluster.py:664 - DEBUG - existing nodes: {u'i-91aefef7': } 2012-05-23 02:33:36,158 PID: 4549 cluster.py:667 - DEBUG - updating existing node i-91aefef7 in self._nodes 2012-05-23 02:33:36,159 PID: 4549 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-23 02:33:37,002 PID: 4549 node.py:733 - INFO - Detaching volume vol-a759adc9 from master 2012-05-23 02:33:38,709 PID: 4549 cluster.py:664 - DEBUG - existing nodes: {u'i-91aefef7': } 2012-05-23 02:33:38,710 PID: 4549 cluster.py:667 - DEBUG - updating existing node i-91aefef7 in self._nodes 2012-05-23 02:33:38,710 PID: 4549 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-23 02:33:38,710 PID: 4549 node.py:828 - INFO - Terminating node: master (i-91aefef7) 2012-05-23 02:33:40,569 PID: 4549 cluster.py:1368 - INFO - Canceling spot instance request: sir-158bb214 2012-05-23 02:33:43,262 PID: 4549 cluster.py:1140 - INFO - Waiting for cluster to terminate... 2012-05-23 02:55:56,167 PID: 71274 config.py:551 - DEBUG - Loading config 2012-05-23 02:55:56,167 PID: 71274 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-23 02:55:56,175 PID: 71274 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-23 11:06:56,085 PID: 521 config.py:551 - DEBUG - Loading config 2012-05-23 11:06:56,085 PID: 521 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-23 11:06:56,088 PID: 521 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-23 11:07:02,950 PID: 521 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-23 11:07:02,950 PID: 521 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-23 11:07:04,800 PID: 521 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-23 11:07:04,800 PID: 521 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-23 11:07:04,801 PID: 521 cli.py:279 - ERROR - Cluster 'mysfmcluster' already exists. Either choose a different tag name, or terminate the existing cluster using: $ starcluster terminate mysfmcluster If you encountered an issue while starting or using 'mysfmcluster' you can reboot and reconfigure the cluster using the 'restart' command: $ starcluster restart mysfmcluster This will reboot all existing nodes and completely reconfigure the cluster without wasting instance hours. 2012-05-23 11:07:56,112 PID: 529 config.py:551 - DEBUG - Loading config 2012-05-23 11:07:56,112 PID: 529 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-23 11:07:56,113 PID: 529 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-23 11:08:01,632 PID: 529 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-23 11:08:01,632 PID: 529 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-23 11:08:23,064 PID: 535 config.py:551 - DEBUG - Loading config 2012-05-23 11:08:23,064 PID: 535 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-23 11:08:23,066 PID: 535 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-23 11:08:27,991 PID: 535 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-23 11:08:27,992 PID: 535 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-23 11:08:48,937 PID: 535 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-23 11:08:48,937 PID: 535 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-23 11:08:49,855 PID: 535 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-23 11:08:49,856 PID: 535 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-23 11:08:52,971 PID: 535 cluster.py:1140 - INFO - Waiting for cluster to terminate... 2012-05-23 11:08:55,156 PID: 535 cluster.py:1377 - INFO - Removing @sc-mysfmcluster placement group 2012-05-23 11:08:56,306 PID: 535 cluster.py:1380 - INFO - Removing @sc-mysfmcluster security group 2012-05-23 11:13:16,843 PID: 556 config.py:551 - DEBUG - Loading config 2012-05-23 11:13:16,843 PID: 556 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-23 11:13:16,846 PID: 556 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-23 11:13:19,992 PID: 556 start.py:176 - INFO - Using default cluster template: mysfmcluster 2012-05-23 11:13:19,993 PID: 556 cluster.py:1539 - INFO - Validating cluster template settings... 2012-05-23 11:13:30,739 PID: 556 cluster.py:1555 - INFO - Cluster template settings are valid 2012-05-23 11:13:30,739 PID: 556 cluster.py:1427 - INFO - Starting cluster... 2012-05-23 11:13:30,740 PID: 556 cluster.py:952 - INFO - Launching a 1-node cluster... 2012-05-23 11:13:30,740 PID: 556 cluster.py:1004 - INFO - Launching master node (ami: ami-fae74193, type: cg1.4xlarge)... 2012-05-23 11:13:31,689 PID: 556 awsutils.py:165 - INFO - Creating security group @sc-mysfmcluster... 2012-05-23 11:13:39,646 PID: 556 awsutils.py:275 - INFO - Creating placement group @sc-mysfmcluster... 2012-05-23 11:13:43,027 PID: 556 cluster.py:772 - INFO - SpotInstanceRequest:sir-eeb33011 2012-05-23 11:13:43,027 PID: 556 cluster.py:1235 - INFO - Waiting for cluster to come up... (updating every 30s) 2012-05-23 11:13:44,257 PID: 556 cluster.py:1165 - INFO - Waiting for open spot requests to become active... 2012-05-23 11:19:55,968 PID: 556 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-23 11:19:55,968 PID: 556 cluster.py:672 - DEBUG - adding node i-f5bce093 to self._nodes list 2012-05-23 11:19:58,779 PID: 556 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-23 11:19:58,779 PID: 556 cluster.py:1193 - INFO - Waiting for all nodes to be in a 'running' state... 2012-05-23 11:19:59,447 PID: 556 cluster.py:664 - DEBUG - existing nodes: {u'i-f5bce093': } 2012-05-23 11:19:59,448 PID: 556 cluster.py:667 - DEBUG - updating existing node i-f5bce093 in self._nodes 2012-05-23 11:19:59,448 PID: 556 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-23 11:19:59,449 PID: 556 cluster.py:1211 - INFO - Waiting for SSH to come up on all nodes... 2012-05-23 11:20:00,268 PID: 556 cluster.py:664 - DEBUG - existing nodes: {u'i-f5bce093': } 2012-05-23 11:20:00,269 PID: 556 cluster.py:667 - DEBUG - updating existing node i-f5bce093 in self._nodes 2012-05-23 11:20:00,269 PID: 556 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-23 11:20:01,245 PID: 556 __init__.py:75 - DEBUG - loading private key /Users/mhelal/.ssh/mhelal.rsa 2012-05-23 11:20:01,247 PID: 556 __init__.py:167 - DEBUG - Using private key /Users/mhelal/.ssh/mhelal.rsa (rsa) 2012-05-23 11:20:01,247 PID: 556 __init__.py:97 - DEBUG - connecting to host ec2-23-20-139-233.compute-1.amazonaws.com on port 22 as user root 2012-05-23 11:21:05,190 PID: 556 cluster.py:664 - DEBUG - existing nodes: {u'i-f5bce093': } 2012-05-23 11:21:05,190 PID: 556 cluster.py:667 - DEBUG - updating existing node i-f5bce093 in self._nodes 2012-05-23 11:21:05,190 PID: 556 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-23 11:21:06,726 PID: 556 __init__.py:97 - DEBUG - connecting to host ec2-23-20-139-233.compute-1.amazonaws.com on port 22 as user root 2012-05-23 11:21:57,189 PID: 556 cluster.py:664 - DEBUG - existing nodes: {u'i-f5bce093': } 2012-05-23 11:21:57,189 PID: 556 cluster.py:667 - DEBUG - updating existing node i-f5bce093 in self._nodes 2012-05-23 11:21:57,190 PID: 556 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-23 11:21:58,029 PID: 556 __init__.py:97 - DEBUG - connecting to host ec2-23-20-139-233.compute-1.amazonaws.com on port 22 as user root 2012-05-23 11:22:04,180 PID: 556 __init__.py:186 - DEBUG - creating sftp connection 2012-05-23 11:22:06,994 PID: 556 utils.py:93 - INFO - Waiting for cluster to come up took 8.399 mins 2012-05-23 11:22:06,994 PID: 556 cluster.py:1454 - INFO - The master node is ec2-23-20-139-233.compute-1.amazonaws.com 2012-05-23 11:22:06,994 PID: 556 cluster.py:1455 - INFO - Setting up the cluster... 2012-05-23 11:22:08,025 PID: 556 cluster.py:1285 - INFO - Attaching volume vol-69bd4807 to master node on /dev/sdz ... 2012-05-23 11:22:09,057 PID: 556 cluster.py:1287 - DEBUG - resp = attaching 2012-05-23 11:22:17,382 PID: 556 cluster.py:664 - DEBUG - existing nodes: {u'i-f5bce093': } 2012-05-23 11:22:17,382 PID: 556 cluster.py:667 - DEBUG - updating existing node i-f5bce093 in self._nodes 2012-05-23 11:22:17,382 PID: 556 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-23 11:22:17,382 PID: 556 clustersetup.py:90 - INFO - Configuring hostnames... 2012-05-23 11:22:17,387 PID: 556 threadpool.py:135 - DEBUG - unfinished_tasks = 1 2012-05-23 11:22:18,388 PID: 556 threadpool.py:135 - DEBUG - unfinished_tasks = 1 2012-05-23 11:22:23,558 PID: 556 clustersetup.py:283 - INFO - Mounting EBS volume vol-69bd4807 on /home... 2012-05-23 11:22:24,249 PID: 556 __init__.py:543 - DEBUG - /dev/sda1 on / type ext4 (rw) 2012-05-23 11:22:24,250 PID: 556 __init__.py:543 - DEBUG - proc on /proc type proc (rw,noexec,nosuid,nodev) 2012-05-23 11:22:24,250 PID: 556 __init__.py:543 - DEBUG - sysfs on /sys type sysfs (rw,noexec,nosuid,nodev) 2012-05-23 11:22:24,251 PID: 556 __init__.py:543 - DEBUG - fusectl on /sys/fs/fuse/connections type fusectl (rw) 2012-05-23 11:22:24,251 PID: 556 __init__.py:543 - DEBUG - none on /sys/kernel/debug type debugfs (rw) 2012-05-23 11:22:24,251 PID: 556 __init__.py:543 - DEBUG - none on /sys/kernel/security type securityfs (rw) 2012-05-23 11:22:24,251 PID: 556 __init__.py:543 - DEBUG - udev on /dev type devtmpfs (rw,mode=0755) 2012-05-23 11:22:24,252 PID: 556 __init__.py:543 - DEBUG - devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) 2012-05-23 11:22:24,252 PID: 556 __init__.py:543 - DEBUG - tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755) 2012-05-23 11:22:24,252 PID: 556 __init__.py:543 - DEBUG - none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880) 2012-05-23 11:22:24,252 PID: 556 __init__.py:543 - DEBUG - none on /run/shm type tmpfs (rw,nosuid,nodev) 2012-05-23 11:22:24,253 PID: 556 __init__.py:543 - DEBUG - /dev/xvdb on /mnt type ext3 (rw) 2012-05-23 11:22:25,902 PID: 556 __init__.py:284 - DEBUG - new /etc/fstab after removing regex ( /home ) matches: LABEL=cloudimg-rootfs / ext4 defaults 0 0 /dev/xvdb /mnt auto defaults,nobootwait,comment=cloudconfig 0 2 2012-05-23 11:22:36,424 PID: 556 clustersetup.py:154 - INFO - Creating cluster user: None (uid: 1001, gid: 1001) 2012-05-23 11:22:36,425 PID: 556 threadpool.py:135 - DEBUG - unfinished_tasks = 1 2012-05-23 11:22:37,426 PID: 556 threadpool.py:135 - DEBUG - unfinished_tasks = 1 2012-05-23 11:22:38,427 PID: 556 threadpool.py:135 - DEBUG - unfinished_tasks = 1 2012-05-23 11:22:38,882 PID: 556 clustersetup.py:171 - DEBUG - user sgeadmin does not exist, creating... 2012-05-23 11:22:39,427 PID: 556 threadpool.py:135 - DEBUG - unfinished_tasks = 1 2012-05-23 11:22:40,428 PID: 556 threadpool.py:135 - DEBUG - unfinished_tasks = 1 2012-05-23 11:22:41,429 PID: 556 clustersetup.py:200 - INFO - Configuring scratch space for user(s): sgeadmin 2012-05-23 11:22:41,430 PID: 556 threadpool.py:135 - DEBUG - unfinished_tasks = 1 2012-05-23 11:22:42,431 PID: 556 threadpool.py:135 - DEBUG - unfinished_tasks = 1 2012-05-23 11:22:43,432 PID: 556 threadpool.py:135 - DEBUG - unfinished_tasks = 1 2012-05-23 11:22:44,432 PID: 556 clustersetup.py:209 - INFO - Configuring /etc/hosts on each node 2012-05-23 11:22:44,433 PID: 556 threadpool.py:135 - DEBUG - unfinished_tasks = 1 2012-05-23 11:22:45,434 PID: 556 threadpool.py:135 - DEBUG - unfinished_tasks = 1 2012-05-23 11:22:45,948 PID: 556 __init__.py:284 - DEBUG - new /etc/hosts after removing regex (master) matches: 127.0.0.1 ubuntu # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts # Added by cloud-init 127.0.1.1 ip-10-16-17-238.ec2.internal ip-10-16-17-238 2012-05-23 11:22:46,435 PID: 556 threadpool.py:135 - DEBUG - unfinished_tasks = 1 2012-05-23 11:22:47,436 PID: 556 threadpool.py:135 - DEBUG - unfinished_tasks = 1 2012-05-23 11:22:48,436 PID: 556 threadpool.py:135 - DEBUG - unfinished_tasks = 1 2012-05-23 11:22:49,437 PID: 556 threadpool.py:135 - DEBUG - unfinished_tasks = 1 2012-05-23 11:22:50,438 PID: 556 threadpool.py:135 - DEBUG - unfinished_tasks = 1 2012-05-23 11:22:51,439 PID: 556 threadpool.py:135 - DEBUG - unfinished_tasks = 1 2012-05-23 11:22:52,440 PID: 556 node.py:606 - INFO - Starting NFS server on master 2012-05-23 11:22:53,392 PID: 556 __init__.py:543 - DEBUG - Rather than invoking init scripts through /etc/init.d, use the service(8) 2012-05-23 11:22:53,393 PID: 556 __init__.py:543 - DEBUG - utility, e.g. service portmap start 2012-05-23 11:22:53,393 PID: 556 __init__.py:543 - DEBUG - 2012-05-23 11:22:53,393 PID: 556 __init__.py:543 - DEBUG - Since the script you are attempting to invoke has been converted to an 2012-05-23 11:22:53,394 PID: 556 __init__.py:543 - DEBUG - Upstart job, you may also use the start(8) utility, e.g. start portmap 2012-05-23 11:22:54,416 PID: 556 __init__.py:540 - DEBUG - command 'mount -t rpc_pipefs sunrpc /var/lib/nfs/rpc_pipefs/' failed with status 32 2012-05-23 11:22:54,416 PID: 556 __init__.py:543 - DEBUG - mount: unknown filesystem type 'rpc_pipefs' 2012-05-23 11:22:55,784 PID: 556 __init__.py:543 - DEBUG - * Exporting directories for NFS kernel daemon... 2012-05-23 11:22:55,784 PID: 556 __init__.py:543 - DEBUG - ...done. 2012-05-23 11:22:55,785 PID: 556 __init__.py:543 - DEBUG - * Starting NFS kernel daemon 2012-05-23 11:22:55,785 PID: 556 __init__.py:543 - DEBUG - ...done. 2012-05-23 11:22:55,785 PID: 556 __init__.py:543 - DEBUG - exportfs: scandir /etc/exports.d: No such file or directory 2012-05-23 11:22:55,785 PID: 556 __init__.py:543 - DEBUG - 2012-05-23 11:22:56,811 PID: 556 __init__.py:543 - DEBUG - exportfs: scandir /etc/exports.d: No such file or directory 2012-05-23 11:22:56,812 PID: 556 __init__.py:543 - DEBUG - 2012-05-23 11:22:56,812 PID: 556 utils.py:93 - INFO - Setting up NFS took 0.073 mins 2012-05-23 11:22:56,813 PID: 556 clustersetup.py:221 - INFO - Configuring passwordless ssh for root 2012-05-23 11:22:59,770 PID: 556 node.py:417 - DEBUG - Using existing key: /root/.ssh/id_rsa 2012-05-23 11:23:01,091 PID: 556 threadpool.py:123 - INFO - Shutting down threads... 2012-05-23 11:23:01,093 PID: 556 threadpool.py:135 - DEBUG - unfinished_tasks = 6 2012-05-23 11:23:02,135 PID: 556 cli.py:287 - DEBUG - Traceback (most recent call last): File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/cli.py", line 255, in main sc.execute(args) File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/commands/start.py", line 194, in execute validate_running=validate_running) File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/cluster.py", line 1414, in start return self._start(create=create, create_only=create_only) File "", line 2, in _start File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/utils.py", line 87, in wrap_f res = func(*arg, **kargs) File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/cluster.py", line 1437, in _start self.setup_cluster() File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/cluster.py", line 1446, in setup_cluster self._setup_cluster() File "", line 2, in _setup_cluster File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/utils.py", line 87, in wrap_f res = func(*arg, **kargs) File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/cluster.py", line 1460, in _setup_cluster self.cluster_shell, self.volumes) File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/clustersetup.py", line 350, in run self._setup_passwordless_ssh() File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/clustersetup.py", line 225, in _setup_passwordless_ssh auth_conn_key=True) File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/node.py", line 418, in generate_key_for_user key = self.ssh.load_remote_rsa_key(private_key) File "/Library/Python/2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/sshutils/__init__.py", line 210, in load_remote_rsa_key key = ssh.RSAKey(file_obj=rfile) File "build/bdist.macosx-10.6-universal/egg/ssh/rsakey.py", line 48, in __init__ self._from_private_key(file_obj, password) File "build/bdist.macosx-10.6-universal/egg/ssh/rsakey.py", line 167, in _from_private_key data = self._read_private_key('RSA', file_obj, password) File "build/bdist.macosx-10.6-universal/egg/ssh/pkey.py", line 323, in _read_private_key raise PasswordRequiredException('Private key file is encrypted') PasswordRequiredException: Private key file is encrypted 2012-05-23 11:23:02,187 PID: 556 cli.py:165 - ERROR - Oops! Looks like you've found a bug in StarCluster 2012-05-23 11:23:02,188 PID: 556 cli.py:166 - ERROR - Crash report written to: /Users/mhelal/.starcluster/logs/crash-report-556.txt 2012-05-23 11:23:02,188 PID: 556 cli.py:167 - ERROR - Please remove any sensitive data from the crash report 2012-05-23 11:23:02,188 PID: 556 cli.py:168 - ERROR - and submit it to starcluster@mit.edu 2012-05-23 11:24:41,102 PID: 614 config.py:551 - DEBUG - Loading config 2012-05-23 11:24:41,102 PID: 614 config.py:118 - DEBUG - Loading file: /Users/mhelal/.starcluster/config 2012-05-23 11:24:41,104 PID: 614 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None} 2012-05-23 11:24:48,270 PID: 614 cluster.py:664 - DEBUG - existing nodes: {} 2012-05-23 11:24:48,271 PID: 614 cluster.py:672 - DEBUG - adding node i-f5bce093 to self._nodes list 2012-05-23 11:24:48,271 PID: 614 cluster.py:680 - DEBUG - returning self._nodes = [] 2012-05-23 11:24:49,139 PID: 614 node.py:910 - DEBUG - Using native OpenSSH client 2012-05-23 11:24:49,139 PID: 614 node.py:919 - DEBUG - ssh_cmd: ssh -i /Users/mhelal/.ssh/mhelal.rsa root@ec2-23-20-139-233.compute-1.amazonaws.com