Skip to main content

Setting Up Network RAID1 With DRBD On Ubuntu 12.04

A Network software raid1

Setting Up Network RAID1 With DRBD On Ubuntu 12.04

This tutorial shows how to set up network RAID1 with the help of DRBD on two Ubuntu 12.04 systems. DRBD stands for Distributed Replicated Block Device and allows you to mirror block devices over a network (like raid1 with 2 disks). This is useful for high-availability setups because if one node fails, all data is still available from the other node.

1 Preliminary Note

We use two servers (both running the same version of linux in our case Ubuntu 12.04):
  • (IP address
  • (IP address:
Both nodes have an unpartitioned second drive (/dev/sdb) with identical size (X GB) that I want to mirror over the network (network RAID1) with the help of DRBD.
It is important that both nodes can resolve each other, either through DNS or through /etc/hosts/etc/hosts would be a better choice (our 2 servers are on the same switch and local. You can modify /etc/hosts on both nodes as follows:
vi /etc/hosts
Then add these 2 lines
...     server1     server2
You must run all the steps in this tutorial with root privileges, so make sure you're logged in as root:
likle this : sudo su or su - if a user root exist

2 Synchronize Time

It is very important that both nodes have the same time. Therefore install the ntp packages:
apt-get install ntp ntpdate

3 Partition /dev/sdb

Right now, our partitioning looks as follows:
fdisk -l
root@server1:~# fdisk -lDisk /dev/sda: 32.2 GB, 32212254720 bytes255 heads, 63 sectors/track, 3916 cylinders, total 62914560 sectorsUnits = sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk identifier: 0x000712c1   Device Boot      Start         End      Blocks   Id  System/dev/sda1   *        2048      499711      248832   83  Linux/dev/sda2          501758    62912511    31205377    5  Extended/dev/sda5          501760    62912511    31205376   8e  Linux LVMDisk /dev/sdb: 32.2 GB, 32212254720 bytes255 heads, 63 sectors/track, 3916 cylinders, total 62914560 sectorsUnits = sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk identifier: 0x00000000Disk /dev/sdb doesn't contain a valid partition tableDisk /dev/mapper/server1-root: 31.4 GB, 31415336960 bytes255 heads, 63 sectors/track, 3819 cylinders, total 61358080 sectorsUnits = sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk identifier: 0x00000000Disk /dev/mapper/server1-root doesn't contain a valid partition tableDisk /dev/mapper/server1-swap_1: 536 MB, 536870912 bytes255 heads, 63 sectors/track, 65 cylinders, total 1048576 sectorsUnits = sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk identifier: 0x00000000Disk /dev/mapper/server1-swap_1 doesn't contain a valid partition tableroot@server1:~#
As you see, /dev/sdb is not partitioned. We change that now and create one big partition on it, /dev/sdb1:
fdisk /dev/sdb
root@server1:~# fdisk /dev/sdbDevice contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabelBuilding a new DOS disklabel with disk identifier 0xf7ab5969.Changes will remain in memory only, until you decide to write them.After that, of course, the previous content won't be recoverable.Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)Command (m for help): <-- nCommand action   e   extended   p   primary partition (1-4)<-- pPartition number (1-4, default 1): <-- 1First sector (2048-62914559, default 2048): <-- ENTERUsing default value 2048Last sector, +sectors or +size{K,M,G} (2048-62914559, default 62914559): <-- ENTERUsing default value 62914559Command (m for help): <-- tSelected partition 1Hex code (type L to list codes): <-- 83Command (m for help): <-- wThe partition table has been altered!Calling ioctl() to re-read partition table.Syncing disks.root@server1:~#
Now run
fdisk -l
again, and you should find /dev/sdb1 in the output:
root@server1:~# fdisk -lDisk /dev/sda: 32.2 GB, 32212254720 bytes255 heads, 63 sectors/track, 3916 cylinders, total 62914560 sectorsUnits = sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk identifier: 0x000712c1   Device Boot      Start         End      Blocks   Id  System/dev/sda1   *        2048      499711      248832   83  Linux/dev/sda2          501758    62912511    31205377    5  Extended/dev/sda5          501760    62912511    31205376   8e  Linux LVMDisk /dev/sdb: 32.2 GB, 32212254720 bytes64 heads, 51 sectors/track, 19275 cylinders, total 62914560 sectorsUnits = sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk identifier: 0xf7ab5969   Device Boot      Start         End      Blocks   Id  System/dev/sdb1            2048    62914559    31456256   83  LinuxDisk /dev/mapper/server1-root: 31.4 GB, 31415336960 bytes255 heads, 63 sectors/track, 3819 cylinders, total 61358080 sectorsUnits = sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk identifier: 0x00000000Disk /dev/mapper/server1-root doesn't contain a valid partition tableDisk /dev/mapper/server1-swap_1: 536 MB, 536870912 bytes255 heads, 63 sectors/track, 65 cylinders, total 1048576 sectorsUnits = sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk identifier: 0x00000000Disk /dev/mapper/server1-swap_1 doesn't contain a valid partition tableroot@server1:~#

4 Install And Configure DRBD

server1/server2: Now install DRBD on both nodes as follows: apt-get install drbd8-utils
Load the DRBD kernel module: modprobe drbd
To check if it is loaded, run:
lsmod | grep drbd
Output should be similar to this one:
root@server1:~#lsmod|grep drbddrbd 273002  0lru_cache              14896  1 drbdroot@server1:~#
Now we back up the original /etc/drbd.conf file and create a new one on both nodes as follows:
cp /etc/drbd.conf /etc/drbd.conf_origcat /dev/null > /etc/drbd.confvi /etc/drbd.conf

global { usage-count no; }
common { syncer { rate 100M; } }
resource r0 {
        protocol C;
        startup {
                wfc-timeout  15;
                degr-wfc-timeout 60;
        net {
                cram-hmac-alg sha1;
                shared-secret "secret";
        on {
                device /dev/drbd0;
                disk /dev/sdb1;
                meta-disk internal;
        on {
                device /dev/drbd0;
                disk /dev/sdb1;
                meta-disk internal;
Make sure you use the correct node names in the file (instead of and - please make sure you use the node names that the command
uname -n
shows on both nodes. Also make sure you fill in the correct IP addresses in the address ;lines and the correct disk in the disk lines (if you don't use /dev/sdb1).
Now we initialize the meta data storage. On both nodes run:
drbdadm create-md r0
root@server1:~# drbdadm create-md r0Writing meta data...initializing activity logNOT initialized bitmapNew drbd meta data block successfully created.root@server1:~#
Then start DRBD on both nodes:
/etc/init.d/drbd start
root@server1:~# /etc/init.d/drbd start * Starting DRBD resources [ d(r0) s(r0) n(r0) ] [ OK ]root@server1:~#
The next step has to be carried out on server1 only:
Now make server1 the primary node:
drbdadm -- --overwrite-data-of-peer primary all
Afterwards, data will start to synchronize between server1 and server2.
Take a look at
cat /proc/drbd
to see the synchronization progress:
root@server2:~# cat /proc/drbdversion: 8.3.11 (api:88/proto:86-96)srcversion: DA5A13F16DE6553FC7CE9B2 0: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r----- ns:0 nr:10166400 dw:10166400 dr:0 al:0 bm:620 lo:1 pe:7407 ua:0 ap:0 ep:1 wo:f oos:21288860 [=====>..............] sync'ed: 32.4% (20788/30716)Mfinish: 0:03:53 speed: 91,180 (86,152) want: 102,400 K/secroot@server2:~#
(You can run
watch cat /proc/drbd
to get an ongoing output of the process. To leave watch, press CTRL+C.)
Wait until the synchronization has finished - output should be as follows:
root@server2:~# cat /proc/drbdversion: 8.3.11 (api:88/proto:86-96)srcversion: DA5A13F16DE6553FC7CE9B2 0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----- ns:0 nr:31455260 dw:31455260 dr:0 al:0 bm:1909 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0root@server2:~#
The snippet ro:Secondary/Primary tells you that this node is the secondary node.
On server1, the output of
cat /proc/drbd

root@server1:~# cat /proc/drbdversion: 8.3.11 (api:88/proto:86-96)srcversion: DA5A13F16DE6553FC7CE9B2 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:31455260 nr:0 dw:0 dr:31455924 al:0 bm:1920 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0root@server1:~#
The snippet ro:Primary/Secondary tells you that this is the primary node.
Now that we have our new network RAID1 block device /dev/drbd0 (which consists of /dev/sdb1 from server1 and server2), let's create an ext4 filesystem on it and mount it to the/data directory. This has to be done only on server1!
mkfs.ext4 /dev/drbd0mkdir /datamount /dev/drbd0 /data
Afterwards you should see /dev/drbd0 in the outputs of...
root@server1:~# mount/dev/mapper/server1-root on / type ext4 (rw,errors=remount-ro)proc on /proc type proc (rw,noexec,nosuid,nodev)sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)fusectl on /sys/fs/fuse/connections type fusectl (rw)none on /sys/kernel/debug type debugfs (rw)none on /sys/kernel/security type securityfs (rw)udev on /dev type devtmpfs (rw,mode=0755)devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)none on /run/shm type tmpfs (rw,nosuid,nodev)/dev/sda1 on /boot type ext2 (rw)/dev/drbd0 on /data type ext4 (rw)root@server1:~#
... and:
df -h
root@server1:~# df -hFilesystem Size Used Avail Use% Mounted on/dev/mapper/server1-root 29G 1017M 27G 4% /udev 238M 4.0K 238M 1% /devtmpfs 99M 228K 99M 1% /runnone 5.0M 4.0K 5.0M 1% /run/locknone 247M 0 247M 0% /run/shm/dev/sda1 228M 24M 193M 11% /boot/dev/drbd0 30G 172M 28G 1% /dataroot@server1:~#

5 Test

Now let's create some files or directories in the /data directory and check whether they get replicated to server2.
touch /data/test1.txttouch /data/test2.txt
ls -l /data/
root@server1:~# ls -l /data/total 16drwx------ 2 root root 16384 2011-10-28 14:12 lost+found-rw-r--r-- 1 root root     0 2011-10-28 14:13 test1.txt-rw-r--r-- 1 root root     0 2011-10-28 14:13 test2.txtroot@server1:~#
Now let's unmount the /data directory on server1:
umount /data
Then assign the secondary role to server1:
drbdadm secondary r0
Now we go to server2, make it the primary node and check if we can see the files/directories we created on server1 in the /data directory on server2.
First we assign the primary role to server2:
drbdadm primary r0
Check the output of
cat /proc/drbd
... and you should see that server2 is the primary node now:
root@server2:~# cat /proc/drbdversion: 8.3.11 (api:88/proto:86-96)srcversion: DA5A13F16DE6553FC7CE9B2 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----    ns:0 nr:31691444 dw:31691444 dr:664 al:0 bm:1909 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0root@server2:~#
Next we create the /data directory and mount /dev/drbd0 to it:
mkdir /datamount /dev/drbd0 /data
Let's check the contents of the /data directory:
ls -l /data/
If everything went fine, it should contain the files/directories that we created on server1:
root@server2:~# ls -l /data/total 16drwx------ 2 root root 16384 2011-10-28 14:12 lost+found-rw-r--r-- 1 root root     0 2011-10-28 14:13 test1.txt-rw-r--r-- 1 root root     0 2011-10-28 14:13 test2.txtroot@server2:~#
Now that we have switched roles, the output of
cat /proc/drbd
on server1 should show you that server1 has the secondary role:
root@server1:~# cat /proc/drbdversion: 8.3.11 (api:88/proto:86-96)srcversion: DA5A13F16DE6553FC7CE9B2 0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----    ns:31691444 nr:185568 dw:421752 dr:31457005 al:83 bm:1920 lo:1 pe:0 ua:0 ap:0 ep:1 wo:f oos:0root@server1:~#

6 Links


  1. This comment has been removed by the author.

  2. mới sinh nên ăn gìHầu hết sản phụ đều mất khá nhiều máu trong quá trình “bể chum” viên uống collagen loại nào tốtDa chảy xệ, kém đàn hồi. Bạn hãy dùng ngón tay cái và ngón trỏ kẹp một mảng da trong vài giây rồi thả ra. cách nấu cháo dinh dưỡng cho bé 7 thángCác mẹ cũng có thể đun sôi gạo từ buổi tối, để đến sáng hôm sau khỏi mất thời gian dậy sớm để chuẩn bị một nồi cháo trắng nhừ nữa. thực đơn cháo dinh dưỡng cho bé 6 tháng tuổi- Ở tuổi này ăn 2-3 bát bột 1 ngày và cộng thêm 600-700ml sữa. lúc nào thì nên bổ sung collagen Viên uống collagen plus làm tăng cường và nuôi dưỡng dây chằng, gân, xương, nướu, mắt, răng và mạch máu. venus pure collagen dạng viên- Tăng cường tính đàn hồi và độ săn chắc của da. avalon collagen drinkVới làn da, ngoài nhiệm vụ liên kết nó còn có nhiệm vụ tạo sự đàn hồi. hanamai collagen gia bao nhieuCó thể do biếng ăn hay do tuổi tác hoặc lý do ngoại cảnh tác động đến, Việc thiếu collagen sẽ đem đến những tác hại khôn lường. uống nhiều collagen shiseido có tốt khôngThường thì sẹo được hình thành la do sợi chằng dọc của da(Elastine) và sợi chằng ngang và mối nối của da (collagen) bị gãy và tổn thương sữa chua cho trẻ em loại nào tốtBạn cần hướng dẫn trẻ súc miệng kỹ càng bằng nước sôi ấm hoặc nước muối loãng cho sạch vì do các vi khuẩn có lợi trong sữa chua hoạt động rất mạnh nên cũng rất dễ làm hỏng men răng, nhất là răng của bé còn khá non nớt nữa đấy.


Post a Comment

Pascal Fares and Open Source Lebanese Movement >

Popular posts from this blog

Setting up MySQL SSL and secure connections

There are different articles on how to setup MySQL with SSL but it’s sometimes difficult to end up with a good simple one. Usually, setting up MySQL SSL is not really a smooth process due to such factors like “it’s not your day”, something is broken apparently or the documentation lies... Read this article : Setting up MySQL SSL and secure connections Pre-requisite : Creating SSL Certificates and Keys Using openssl

HOWTO remove all dangling commits from your git repository

A good explanation of the dangling ( fr: ballants) commits source tells you how they get created. git fsck --full   Checking object directories: 100% (300/300), done. Checking objects: 100% (10658/10658), done. dangling commit x.... dangling blob y.... dangling commit z.... dangling blob w.... dangling blob a.... dangling commit b.... How to quickly remove those? git reflog expire --expire=now --all git gc --prune=now