curl - fetch headers of URL

To fetch only the headers of a HTTP URL, we can use the curl with the -I switch. Consider the following command line:

$ curl -I https://www.stanford.edu

HTTP/2 200 

server: nginx

date: Sun, 02 Dec 2018 10:09:19 GMT

content-type: text/html; charset=UTF-8

set-cookie: wordpress_google_apps_login=3612792336cb3b1361acd56bb0addf07; path=/; secure

link: <https://www.stanford.edu/wp-json/>; rel="https://api.w.org/"

link: <https://www.stanford.edu/>; rel=shortlink

x-frame-options: SAMEORIGIN

vary: Accept-Encoding

wpe-backend: apache

x-wpe-loopback-upstream-addr: 127.0.0.1:6783

x-cacheable: NO:Set Known Cookie

cache-control: max-age=0, must-revalidate, private

x-cache: MISS

x-pass-why: 

x-cache-group: normal

x-type: default

$ 

In this we fetch the headers sent from the URL of the stanford university. 

Ubuntu - Mount a ephemeral storage on AWS EC2 machine.

Consider a AWS EC2 machine which is accompanied by an ephemeral storage. Many times, this ephemeral storage is not visible to us. In such a situation, we have search for it and set it up manually. Following are the steps taken by me to identify and mount ephemeral storage on a EC2 machine.


Step 1: Identify the disk which needs mounting . For this we run the fdisk command:
ubuntu@ip-XXX-XXX-XXX-XXX:~$ sudo fdisk -l
Disk /dev/loop0: 87.9 MiB, 92164096 bytes, 180008 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop1: 12.7 MiB, 13324288 bytes, 26024 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/nvme0n1: 8 GiB, 8589934592 bytes, 16777216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x9f3e4931

Device         Boot Start      End  Sectors Size Id Type
/dev/nvme0n1p1 *     2048 16777182 16775135   8G 83 Linux


Disk /dev/nvme1n1: 46.6 GiB, 50000000000 bytes, 97656250 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
ubuntu@ip-XXX-XXX-XXX-XXX:~$

Step 2: Once we have identfied our ephemeral storage which we want to mount. Next we identify the folder on which we want to mount it on. Since,we have decided to mount it on /opt . Now we will proceed with the following steps:

ubuntu@ip-XXX-XXX-XXX-XXX:/$ sudo mkdir opt

Step 3: Create filesystem on your volume (make sure you choose the correct volume because this creates a new file system on the volume):

ubuntu@ip-XXX-XXX-XXX-XXX:/$ sudo mkfs.ext4 /dev/nvme1n1
mke2fs 1.44.1 (24-Mar-2018)
Discarding device blocks: done                            
Creating filesystem with 12207031 4k blocks and 3055616 inodes
Filesystem UUID: 1e7f28c2-19b3-4a94-8689-4d9b75496eff
Superblock backups stored on blocks: 
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
4096000, 7962624, 11239424

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (65536 blocks): done
Writing superblocks and filesystem accounting information: done   

Step 4: Let us now, mount our volume:
ubuntu@ip-XXX-XXX-XXX-XXX:/$ sudo mount -t ext4 /dev/nvme1n1 /opt
ubuntu@ip-XXX-XXX-XXX-XXX:/$ mount -a
mount: only root can use "--all" option
ubuntu@ip-XXX-XXX-XXX-XXX:/$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            1.8G     0  1.8G   0% /dev
tmpfs           371M  716K  370M   1% /run
/dev/nvme0n1p1  7.7G  1.1G  6.7G  14% /
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/loop0       88M   88M     0 100% /snap/core/5328
/dev/loop1       13M   13M     0 100% /snap/amazon-ssm-agent/495
tmpfs           371M     0  371M   0% /run/user/1000
/dev/nvme1n1     46G   53M   44G   1% /opt
ubuntu@ip-XXX-XXX-XXX-XXX:/$ 


Step 5: If you want to preserve the mount after e.g. a restart, open /etc/fstab and add the mount to it

ubuntu@ip-XXX-XXX-XXX-XXX:/$ echo "/dev/nvme1n1 /opt auto noatime 0 0" | sudo tee -a /etc/fstab
/dev/nvme1n1 /opt auto noatime 0 0
ubuntu@ip-XXX-XXX-XXX-XXX:/$ 

Step 6: Make sure nothing is wrong with fstab by mounting all
ubuntu@ip-XXX-XXX-XXX-XXX:/$ sudo mount -a
ubuntu@ip-XXX-XXX-XXX-XXX:/$ 

Step 7: Check for the available disk size:
ubuntu@ip-XXX-XXX-XXX-XXX:/opt$ df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/nvme1n1 46G 53M 44G 1% /opt
ubuntu@ip-XXX-XXX-XXX-XXX:/opt$ ls
lost+found
ubuntu@ip-XXX-XXX-XXX-XXX:/opt$ ls -l
total 16
drwx------ 2 root root 16384 Nov 13 08:45 lost+found
ubuntu@ip-XXX-XXX-XXX-XXX:/opt$ 





Reference:

• https://serverfault.com/questions/433703/how-to-use-instance-store-volumes-storage-in-amazon-ec2

Python - Machine Learning - Install scikit-learn in Ubuntu

To install scikit-learn package, we need to install its dependencies:

$ sudo pip install scipy
The directory '/home/ubuntu/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/ubuntu/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting scipy
  Downloading https://files.pythonhosted.org/packages/2a/f3/de9c1bd16311982711209edaa8c6caa962db30ebb6a8cc6f1dcd2d3ef616/scipy-1.1.0-cp27-cp27mu-manylinux1_x86_64.whl (30.8MB)
    100% |████████████████████████████████| 30.8MB 38kB/s 
Requirement already satisfied: numpy>=1.8.2 in ./.local/lib/python2.7/site-packages (from scipy)
Installing collected packages: scipy
Successfully installed scipy-1.1.0
$


Now Install sklearn:

$ sudo pip install sklearn
The directory '/home/ubuntu/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/ubuntu/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting sklearn
  Downloading https://files.pythonhosted.org/packages/1e/7a/dbb3be0ce9bd5c8b7e3d87328e79063f8b263b2b1bfa4774cb1147bfcd3f/sklearn-0.0.tar.gz
Collecting scikit-learn (from sklearn)
  Downloading https://files.pythonhosted.org/packages/bc/67/370aa248f54769a56216707ad7b9af19745e85a603fafa47bde353f327fb/scikit_learn-0.19.2-cp27-cp27mu-manylinux1_x86_64.whl (5.0MB)
    100% |████████████████████████████████| 5.0MB 179kB/s 
Installing collected packages: scikit-learn, sklearn
  Running setup.py install for sklearn ... done
Successfully installed scikit-learn-0.19.2 sklearn-0.0
$

Refer:

• http://scikit-learn.org/stable/

Julia - Language - Installation in Ubuntu

Visit JuliaLang website and download the current stable release: https://julialang.org/downloads/


Step 1: Open Terminal and download Julia:

$ wget https://julialang-s3.julialang.org/bin/linux/x64/0.6/julia-0.6.0-linux-x86_64.tar.gz

Step 2: Uncompress the Archive:

$ tar -zxvf julia-0.6.0-linux-x86_64.tar.gz

Step 3: After uncompressing the archive - the following folder was created:

$ ls -l
total 1686492
-rw-rw-r-- 1 ubuntu ubuntu   66543012 Jun 20 00:25 julia-0.6.0-linux-x86_64.tar.gz
drwxr-xr-x 7 ubuntu ubuntu       4096 Jun 19 19:28 julia-903644385b
$

Step 4: Move this folder to a convinient location:

$ cd /opt
/opt$

/opt$ mv /home/mhc/Downloads/julia-903644385b .
/opt$

Step 5: Create a softlink for Julia in /usr/bin - this is so that we can run it from any location:

$ cd /usr/bin
/usr/bin$ sudo ln -s /opt/julia-903644385b/bin/julia julia
/usr/bin$

Step 6: Run julia:

$ cd ~
~$ julia
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: https://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.6.0 (2017-06-19 13:05 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release
|__/                   |  x86_64-pc-linux-gnu

julia> 

Ardiuno - Ubuntu - Setting up IDE to work with Ardiuno Uno R3 and run Blink Program

Step 1. Download Arduino IDE: https://downloads.arduino.cc/arduino-1.8.1-linux64.tar.xz


Step 2. Extract the package to: /home/sashank/Documents/Arduino_Projects/Tweak/arduino-1.8.1

Step 3. Run the install.sh:

 ~/Documents/Arduino_Projects/Tweak/arduino-1.8.1 $ ./install.sh
Adding desktop shortcut, menu item and file associations for Arduino IDE...touch: cannot touch '/home/sashank/.local/share/applications/mimeapps.list': Permission denied
mv: try to overwrite '/home/sashank/.local/share/applications/mimeapps.list', overriding mode 0644 (rw-r--r--)? y
 done!
 ~/Documents/Arduino_Projects/Tweak/arduino-1.8.1 $

Step 4. Open Arduino IDE.

Step  5. Go through Arduino UNO: https://www.arduino.cc/en/Guide/ArduinoUno

Step 6. Connect your Uno board with an A B USB cable; sometimes this cable is called a USB printer cable

The green power LED (labelled PWR) should go on.

Step 7. Open the Ardiuno IDE . Follow the path ( File -- > Examples -- > 01.Basics --> Blink ) and open the blink program.

Step 8. Click on Verify Button to Verify the program:

Step 9. On verification, we must receive message - Done Compiling in Status Bar:

Step 10. Let us now upload the program to the Ardiuno by clicking on the upload button:




Step 11. After upload we have received error message as follows:

Arduino: 1.8.1 (Linux), Board: "Arduino/Genuino Uno"

Sketch uses 928 bytes (2%) of program storage space. Maximum is 32256 bytes.
Global variables use 9 bytes (0%) of dynamic memory, leaving 2039 bytes for local variables. Maximum is 2048 bytes.
avrdude: ser_open(): can't open device "/dev/ttyACM0": Permission denied
Problem uploading to board.  See http://www.arduino.cc/en/Guide/Troubleshooting#upload for suggestions.

This report would have more information with
"Show verbose output during compilation"
option enabled in File -> Preferences.

To solve this error, I have referred the below blog:

http://arduino-er.blogspot.in/2014/08/arduino-ide-error-avrdude-seropen-cant.html

Step 12: To fix it, run the following commands at terminal:

sudo usermod -a -G dialout <<username>>
sudo chmod a+rw /dev/ttyACM0

 ~/Documents/Arduino_Projects/Tweak/arduino-1.8.1 $ sudo usermod -a -G dialout sashank
 ~/Documents/Arduino_Projects/Tweak/arduino-1.8.1 $ sudo chmod a+rw /dev/ttyACM0
 ~/Documents/Arduino_Projects/Tweak/arduino-1.8.1 $

Step 13. Upload Again. This time Upload is successful - we have received the following message: Done Uploading

This time received the message:

Sketch uses 928 bytes (2%) of program storage space. Maximum is 32256 bytes.
Global variables use 9 bytes (0%) of dynamic memory, leaving 2039 bytes for local variables. Maximum is 2048 bytes.

The blinker is blinking every 1 second. It was already blinking. Hence let us now modify the program for making it to blink every 3 seconds.


Step 14. Make the program to blink every 3 seconds:

// the setup function runs once when you press reset or power the board
void setup() {
  // initialize digital pin LED_BUILTIN as an output.
  pinMode(LED_BUILTIN, OUTPUT);
}

// the loop function runs over and over again forever
void loop() {
  digitalWrite(LED_BUILTIN, HIGH);   // turn the LED on (HIGH is the voltage level)
  delay(3000);                       // wait for a second
  digitalWrite(LED_BUILTIN, LOW);    // turn the LED off by making the voltage LOW
  delay(3000);                       // wait for a second
}

We find that the orange LED in Arduino is now blinking every 3 seconds:

Success.

Ubuntu - Install Application - Google Web Designer

Step 1: Download Google Web Designer from website: https://www.google.com/webdesigner

 ~/Downloads $ wget https://dl.google.com/linux/direct/google-webdesigner_current_amd64.deb
--2017-03-09 10:40:33--  https://dl.google.com/linux/direct/google-webdesigner_current_amd64.deb
Resolving dl.google.com (dl.google.com)... 216.58.197.78, 2404:6800:4007:800::200e
Connecting to dl.google.com (dl.google.com)|216.58.197.78|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 31455488 (30M) [application/x-debian-package]
Saving to: 'google-webdesigner_current_amd64.deb'

100%[==============================================================================================================================>] 31,455,488  1022KB/s   in 26s    

2017-03-09 10:41:00 (1.14 MB/s) - 'google-webdesigner_current_amd64.deb' saved [31455488/31455488]

 ~/Downloads $

Step 2: Install Google Web Designer using dpkg package manager:

$ sudo dpkg -i google-webdesigner_current_amd64.deb
Selecting previously unselected package google-webdesigner.
(Reading database ... 223100 files and directories currently installed.)
Preparing to unpack google-webdesigner_current_amd64.deb ...
Unpacking google-webdesigner (1.3.10.0-1) ...
Setting up google-webdesigner (1.3.10.0-1) ...
Processing triggers for gnome-menus (3.10.1-0ubuntu2) ...
Processing triggers for desktop-file-utils (0.22-1ubuntu1) ...
Processing triggers for mime-support (3.54ubuntu1.1) ...
 ~/Downloads $

Step 3: Check to confirm installation:

 ~/Downloads $ which google-webdesigner
/usr/bin/google-webdesigner
 ~/Downloads $

Ubuntu - Install / Upgrade Application - Chrome - web browser

 

Step 1: Download the latest version of Google Chrome from website: https://www.google.com/chrome/browser/desktop/

 ~/Downloads $ wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
--2017-03-08 12:13:03--  https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
Resolving dl.google.com (dl.google.com)... 216.58.203.206, 2404:6800:4009:804::200e
Connecting to dl.google.com (dl.google.com)|216.58.203.206|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 45892904 (44M) [application/x-debian-package]
Saving to: 'google-chrome-stable_current_amd64.deb'

100%[==============================================================================================================================>] 45,892,904   590KB/s   in 85s   

2017-03-08 12:14:30 (530 KB/s) - 'google-chrome-stable_current_amd64.deb' saved [45892904/45892904]

 ~/Downloads $

Step 2: To upgrade Chrome: Let us use gdebi to install this package:

 ~/Downloads $ sudo gdebi "google-chrome-stable_current_amd64.deb"
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Building data structures... Done
Building data structures... Done

The web browser from Google
 Google Chrome is a browser that combines a minimal design with sophisticated technology to make the web faster, safer, and easier.
Do you want to install the software package? [y/N]:y
(Reading database ... 223107 files and directories currently installed.)
Preparing to unpack google-chrome-stable_current_amd64 (3).deb ...
Unpacking google-chrome-stable (56.0.2924.87-1) over (51.0.2704.63-1) ...
Setting up google-chrome-stable (56.0.2924.87-1) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Processing triggers for gnome-menus (3.10.1-0ubuntu2) ...
Processing triggers for desktop-file-utils (0.22-1ubuntu1) ...
Processing triggers for mime-support (3.54ubuntu1.1) ...
 ~/Downloads $

Step 3: Check out the version of Chrome Installed:

 ~/Downloads $ google-chrome --version
Google Chrome 56.0.2924.87
 ~/Downloads $

Google Chrome is successfully Upgraded!

Django - 002 - Starting a project - django-admin.py startproject

A project in django is a collection of settings for an instance of Django. This includes:

• Django specific options
• application specific settings
• database configuration

Select the project directory where you want to host your django setup. Now change into that directory.
~ $ cd ~/Documents/my_d3_project/django

Since, we have installed django using pip utility, django-admin.py should be on your system path. You can verify if you can access it:
~/Documents/my_d3_project/django $ which django-admin.py
/opt/anaconda/bin/django-admin.py

This is the path of django-admin.py. Now, run the following command to create a myD3site directory in our current directory.

~/Documents/my_d3_project/django $ django-admin.py startproject myD3site
~/Documents/my_d3_project/django $ ls
myD3site

Let us look inside to find what startproject created. We can use the tree command in Ubuntu:
~/Documents/my_d3_project/django $ tree .
.
`-- myD3site
    |-- manage.py
    `-- myD3site
        |-- __init__.py
        |-- settings.py
        |-- urls.py
        `-- wsgi.py

2 directories, 5 files
~/Documents/my_d3_project/django $ 

The files are:

1. The outer myD3site/ root directory - It is just a container for your project. Its name does not matter to Django, you can rename it to anything you like.
2. manage.py - a command line utility that lets you interact with this django project in various ways
3. The inner myD3site/ directory - It is the python package for your project. It is the name you will use to import anything inside it. (for example, myD3site.urls)
4. myD3site/__init__.py - an empty file that tells Python that this directory should be considered a Python package. This is a file required by Python and tells it to treat this directory as a package (i.e, a group of modules)
5. myD3site/settings.py - settings or configuration for this django project
6. myD3site/urls.py - The URL declarations for this django project: "a table of contents for your django powered site"
7. myD3site/wsgi.py - an entry point for WSGI-compatible web servers to serve your project. 

Django - 001 -Install django using PIP on Ubuntu

Step 1: Ensure that you have the python 2.7 software setup.

Step 2: Install using PIP manager, by running the following comand at the terminal prompt:


 $ sudo pip install Django==1.10.4
Downloading/unpacking Django==1.10.4
  Downloading Django-1.10.4-py2.py3-none-any.whl (6.8MB): 6.8MB downloaded
Installing collected packages: Django
  Found existing installation: Django 1.8.4
    Uninstalling Django:
      Successfully uninstalled Django
Successfully installed Django
Cleaning up...
 $


Step 3: Confirm the installation by importing the django module in Python interpreter:

$ python
Python 2.7.10 |Anaconda 2.3.0 (64-bit)| (default, May 28 2015, 17:02:03) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://binstar.org
>>> import django
>>> django.VERSION
(1, 10, 4, u'final', 0)
>>> exit()
$ 

References:
https://www.djangoproject.com/download/