Migrate Data from S3 to Google Cloud

I noticed that google cloud storage is a bit cheaper than Amazon S3. Although, it seems not much, it can be a lot if you have some big data sets or huge backups stored in Amazon`s storage solution. This tutorial show how you can migrate from/to Google Cloud Storage. But keep in mind that you will have to pay for the file transfair!

You can do that locally or on your cloud shell. I would recommend using some vm on google or amazon, cause cloud shell will terminate your session after some minutes... which is annoying for bigger datasets.

Before you begin, connect to your virtual machine (using SSH) and install the google utils (you can find a tutorial here).

You will also have to install and configure amazons awscli-tool. This can be done link this:

sudo apt install awscli
aws configure

Now you can copy files from s3 to google like this:

gsutil -m rsync -r s3://bucket/path/  gs://anotherbucket

On google cloud shell I still got the following error:

ERROR 0127 10:05:02.767142 utils.py] Unable to read instance data, giving up
Failure: No handler was ready to authenticate. 4 handlers were checked. ['HmacAuthV1Handler', 'DevshellAuth', 'OAuth2Auth', 'OAuth2ServiceAccountAuth'] Check your credentials.

But setting the key & secret manually solved the problem:

export AWS_ACCESS_KEY_ID="aws_access_key_id"
export AWS_SECRET_ACCESS_KEY="aws_secret_access_key"

Links