Monday, September 17, 2012

Backing up with Percona's Xtrabackup

 Percona's backup allows for full and incremental backups and it is open source.  I tried it on a Redhat 6.2 server called Redhat62 and it worked great.   I installed a new version of MySQL and did a restore using files I had backed up using MySQL Enterprise backup and to my surprise innobackupex restored the data and it worked.  The only thing I had to do was change permissions before I started MySQL.

Here is what I did:

Create a backup user for MySQL backups.

mysql>; CREATE USER ’bkpuser’@’localhost’ IDENTIFIED BY ’password’;
mysql>; REVOKE ALL PRIVILEGES, GRANT OPTION FROM ’bkpuser’@'localhost';
mysql>; GRANT RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO ’bkpuser’@’localhost’;
mysql>; FLUSH PRIVILEGES;


To do a full backup of MySQL run this command:
innobackupex --user=bkpuser --password=password --parallel=4 --use-memory=4G /path/to/BACKUP-DIR

My backup directory on Redhat62 is /storage/backups

innobackupex reads the my.cnf file so it knows where the MySQL data is located.  

By Default innobackupex uses only 100MB for running the backup.  To increase the speed of the backup  you need to allocate more memory.  Set the "--use-memory=" to most that  you have free.

You can increase the number of threads by using the parallel parameter.

YOU NEED TO PREPARE THE DATA FOR RESTORE
After the backup is completed you need to run the "--apply-log" before you can restore the data.  The "--apply-log" replays committed transaction in the log files and rolls back uncommitted transactions.

innobackupex  --apply-log /path/to/BACKUP-DIR/time_stamp

To restore you need to run this command:  MYSQL NEEDS TO BE SHUTDOWN DURING THE RESTORE

innobackupex  --copy-back /path/to/BACKUP-DIR

I had previously had databases in the /var/lib/mysql directory with the same name.  innobackupex failed with a message.  I deleted all of the files under /var/lib/mysql and ran it again.  It completed successfully.   I guess since it is doing a full restore it doesn't write over the files that are there.  You will need to clear our all of the files in the directory to which it is restoring.

After the restore is done run this command:

chown -R mysql:mysql /var/lib/mysql


You can do incremental backups with innobackupex.  First you need to do a Full backup and then you can run incrementals.   If you are going to use incremental backups then your Full backups you will need to have the --redo-only option added to it.  This is explained in Pecona's manual:
Preparing an Incremental Backup with innobackupex Preparing incremental backups is a bit different than full
ones. This is, perhaps, the stage where more attention is needed:
• First, only the committed transactions must be replayed on each backup. This will put the base full backup
and the incremental ones altogether.
• Then, the uncommitted transaction must be rolled back in order to have a ready-to-use backup.
If you replay the commit ed transactions and rollback the uncommitted ones on the base backup, you will not be able
to add the incremental ones. If you do this on an incremental one, you won’t be able to add data from that moment
and the remaining increments.
Having this in mind, the procedure is very straight-forward using the --redo-only option, starting with the base
backup:

innobackupex --user=bkpuser --password=password --redo-only /storage/backups

To run an incremental you need to issue this command:

innobackupex  --user=bkpuser --password=password --incremental /storage/backups --incremental-basedir

What if you only want to restore 1 database or 1 table instead of the whole backup?  Here is what you need to do. 

Start up a second session of MySQL on the same server, but point it to your backup directory that is holding the data you want to restore.

To start a second session:
mysqld --basedir=/usr --user=mysql --log-error=/path/to/BACKUP-DIR --open-files-limit=4096 --pid-file=/path/to/BACKUPDIR/Time_Stamp/mysql.pid --port 3307 --datadir=/path/to/BACKUPDIR/Time_Stamp

Here is an example:
mysqld --basedir=/usr  --user=mysql --log-error=/storage/backups/mysql/mysql.log --open-files-limit=4096 --pid-file=/storage/backups/mysql/2012-09-14_10-01-55/mysql.pid --port 3307 --datadir=/storage/backups/mysql/2012-09-14_10-01-55

After you start your second session of MySQL you can now do a MySQLdump the database or table that you want to restore.

To restore a database, type this:
mysqldump -P 3307 -u bkpuser -p password databasename > databasename.sql

To restore a table, type this:
mysqldump -P 3307 -u bkpuser -p password databasename tablename > tablename.sql

I put this information from the following sources: Percona's xtrabackup pdf file found on their website,http://www.tekovic.com/mysql-hot-backup-with-xtrabackup-on-centos and  http://steveswanson.wordpress.com/2009/04/21/exporting-and-importing-an-individual-mysql-table/


Friday, June 15, 2012

Remotely copy hosts file to Windows 7


I have users that use laptops to work remote.  Unfortunately, when they VPN into to our server they can route, but they have problems with DNS.   So for these users I have a hosts file setup with the servers that they need.   Normally, I would just go to their C drive \\computername\c$  and drill down and manually replace it.   That works fine, but since I started using PDQ Deploy - From Admin Arsenal (This is a super Great application - I strongly recommend it) I wanted to see if I could deploy the hosts file to my Windows 7 Professional laptops using a batch file. 

I ran into all kinds of access denied problems.  Everywhere I searched on the Internet for help I ran into you can't it is a security feature in Windows 7.   I tried many things that I found online, but with no success.  That was until I ran into this post: http://stackoverflow.com/questions/10724591/how-to-remote-execute-an-elevated-remote-script-in-powershell.

To be honest, I didn't know what "start-process" was, but since I have googled it.  After putting together my script, I know there more efficient ways to write this script, but for a quick batch file that is easy to follow this is what I wrote.


start-process cmd.exe -verb runas /env /user:username@domainname.com
echo y |copy "\\servername\apps\misc\Windows 7 hosts file\hosts" C:\windows\temp
echo y |copy "\\servername\apps\misc\Windows 7 hosts file\hosts.orig" C:\windows\temp
echo y |copy C:\windows\temp\hosts C:\windows\system32\drivers\etc\hosts
echo y |copy C:\windows\temp\hosts.orig C:\windows\system32\drivers\etc\hosts.orig

Let me explain the script a little bit.   I didn't test, but maybe I could have copied from the network share to C:\windows\system32\drivers\etc directly, but I remember getting an error about copying from a network location, so I decided to copy to the temp directory on the computer and then copy over the original hosts file under the etc folder.   Please experiment or write better code or show me how you would have done this.    Also, I have a copy of an original windows 7 hosts file that I copy into place (the etc folder) so that if I ever need to change it or the user needs to change it while on the road it is there.   

The "echo y" just answers the questions that I do want to overwrite the original file.  I named my batch file copyhost.bat

Now that I have the script written Let's turn to PDQ Deploy and set it up to depoly the hosts file.




If you are already familiar with PDQ Deploy you know how easy it is to setup deploying files.   I just choose my batch file and include the entire directory which includes "copyhost.bat", "hosts", "hosts.orig"  and that it.  I save it and the I just deploy it by choosing deploy and then choosing which computers I want to deploy this to.  Within seconds I'm done and I have a status report saying "successful".

Let me know if you figured out another way to do this.

Tuesday, June 5, 2012

Backup and Restore using mysqlbackup from MySQL Enterprise


All this was done in a lab.  Don't do this on a production server.  There is my warning and disclaimer.

I did a full backup of my MySQL instance which resides under /var/lib/mysql.   I then delete all the data, all the ib* files, and mysql.   After deleting those files I ran a full recovers.  Here is how I did it.

Full Backup
Run the following command to do a full backup.   What this will do is backup everything in /var/lib/mysql and with the backup-and-apply-log it will go back and grab anything that has changed during the backup.   Also, that argument prepares the backup for a restore.
./mysqlbackup --user=root --password  --backup-dir=/storage/backups backup-and-apply-log
 
You need to specify a backup directory that is empty of mysqlbackup will complain.
When the backup completes you will have a message at the end telling you that it was successful.

I cd to the location of my backup and everything looked like it was there.   I ran a du -sh * and results looked the same.

Full Restore
./mysqlbackup --defaults-file=/storage/backups/backup-my.cnf --datadir=/var/lib/mysql --backup-dir=/storage/backups copy-back

Under "--default-file=" I originally put /etc/my.cnf thinking that I should use the my.cnf file I created.  However, part of the backup creates a "backup-my.cnf" with a few lines in it.  You should use this file for the restore or you could put these lines in your restore command.  It is much easier to just use the file that was created.

I started mysql /etc/init.d/mysql start, but it failed with the PID error.   I looked at my restored directory and noticed many of the files was owned by root, so I ran this command on the mysql directory.
chown -R mysql:mysql mysql

Now MySQL starts and every thing looks great.  

Thursday, May 24, 2012

Instead of Samba use Centrify

I've used samba for a long time and love it, but I ran into a product called Centrify that is so easy to setup and use for kerberos authentication against a Windows Active Directory that I don't think I'm going back to samba.

Here is a very brief explanation of how I set it up on my Linux Server.


Download Centrify Suite and the centrify samba package.

1. Uninstall your current ssh and samba.
2. Login as "root"
3. Extract the files.
4. Run "install-express.sh"
5. Answer all the questions.
6.  Reboot.
7.  Install the samba packages.


Make the following changes in /etc/centrify/centrifydc.conf file.
Change the shell from /bin/bash to /bin/ksh
Change the auto.schema.primary.gid: 100
Change the auto.schema.private.group: false

Create a group.ovr file in /etc/centrify/  Add the following line:
+domain users:users:::

Move MySQL to a different Directory


Steps I use to install MySQL to a different directory.   The truth is I just use the rpm and let it install in the default /var/lib/mysql on Redhat and then I do the following to move it to another directory.

After the install I setup my my.cnf file and then start MySQL.   I don't think it is necessary to start it, but I do so that the ib files can be created.  I then stop MySQL.   I cd to /var/lib and move the mysql directory to wherever I want.  In my case I move it to my /data directory which is on a different filesystem.

So I issued this command mv mysql /data
Then I create a symbolic link to my new mysql location by using this command:
ln -s /data/mysql mysql

Now under /var/lib/ I have a symbolic link that points to /data/mysql.   If I cd to /var/lib/mysql I will be in /data/mysql.

One reason I do this is I don't like to have data in my root partition in case I fill it up.  Also, in my setup I don't have enough room in my root partition to host mysql.

After making this change I started to get this error:
starting mysql.the server quit without updating pid file failed.   I've seen this error before and it is usually a permission issue.   I checked my permissions and the permissions were fine.  

The problem is selinux the iptables or firewall.
To check run this command:
semodule -l 

Check and see if mysql is in the list?

Now temporarily disable selinux with this command:

echo > 0 /selinux/enforce

or 

service iptables stop

Now try to start MySQL.  Did it start?  Mine Did.

Now you know the problem is with selinux.  You have a few chooses:

1. you can disable selinux by adding SELINUX=disabled in /etc/selinux/config.  The lease secure, but fastest fix.
2. You can disable the mysql module in selinux.  To disable mysql run this command:

semodule -v -d mysql  or semodule --disable=mysql

To re-enable module run this command:
semodule -v -e mysql or semodule --enable=mysql

The "-v" is verbose and is not necessary.

3. You can keep it enabled, but follow the information in this link:

Tuesday, May 22, 2012

LVM INACTIVE

After a reboot I noticed that my iSCSI LVM was missing.   I did an lvscan and it was listed, but as INACTIVE.

I ran lvchange -a y /dev/vg_nobakcup/lg_nobackup and the logical volume became active again.
I ran a lvscan to verify it was active.  
I then mounted the volume.
Upon reboot I had the same problem, so I added lvchange -a y /dev/vg_nobackup/lg_nobackup and rebooted again.   This time the volume was active and mounted.  PROBLEM SOLVED.

I believe the issue is this is an iSCSI Volume connecting to Microsoft's iSCSI Storage server.  I believe the connection just times out before connecting.  

Installing Redhat 6.2 & Linux Integration Services on Hyper-V

I installed Redhat 6.2 today.  I added Linux integration services 3.2 and after I added the service I no longer had a cdrom drive or my networks.   So this time I'm trying a little different approach.  I believe this is what I did the first time and it worked.
Before you install Redhat 6.2 add a legacy network card to your settings.   Redhat will recognized the legacy network adapter, but not the normal network adapter.

Once you have configured Redhat and it has rebooted you will have a desktop, but you won't be able to do anything with the desktop because you don't have mouse or keyboard control.  You need to install the Linux Integration Services.   Press "CTRL+ALT+F2" at the same time.   This will will give you a terminal window.  Login as root.

In Hyper-V go to Media under Virtual Machine Connection and attach the iso file as a DVD drive.

Once it is attached use the mount command on Linux to mount the DVD.

mount /dev/cdrom /media
cd to /media
run install.sh
reboot
After the reboot the clock will work, the mouse will work, the keyboard will work, the network adapter will work.   
Now you ready to use your new Redhat 6.2.
After Installing Linux Integration Services 3.2 you will no longer have a cdrom.
You will need to run this command:
insmod /lib/modules/$(uname -r)/kernel/drivers/ata/ata_piix.ko 
for the cdrom to mount.

Wednesday, May 16, 2012

Adding LVM from command line

This is how you setup a Physical Disk, Volume Group and a Logical Volume using the command line in Linux.

This example will be from a Redhat Enterprise 6.2 Installation.

So we will create a Physical Disk, Physical Volume, then a Volume Group, then a Logical Volume and then mount the volume.  Type only what is in "red".
 


Step #1 - Find out what device your new hard drive is assigned.

fdisk -l 

You should see all your disk including the one that you just added.  Here is the one that I just added.  

Disk /dev/sdb: 343.6 GB, 343597383680 bytes
255 heads, 63 sectors/track, 41773 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1       41774   335544319+  8e  Linux LVM

Our new device is called "/dev/sdb1"

Step #2 - Partition the disk with fdisk. (Create Physical Disk)

fdisk /dev/sdb1
m - to see all the command options
n - to add the new partition
p - to make it a primary partition
1 - to make it the 1st partition
First cylinder (1-41773, default 1):  PRESS ENTER
Last cylinder, +cylinders or +size{K,M,G} (1-41773, default 41773): PRESS ENTER

***NOW WE NEED change the partition's system id
t - to change the partition's system type
L - to list all the system type
FOR LVM type 8e
8e - for LVM PRESS ENTER
w - to write table to disk and exit.

STEP #3 - Create a Physical Volume with pvcreate.

pvscan - will show you the physical volumes.
PV /dev/sda2   VG vg_redhat62   lvm2 [69.51 GiB / 0    free]
  Total: 1 [69.51 GiB] / in use: 1 [69.51 GiB] / in no VG: 0 [0   ]

As you can see our new physical disk does not show up.  It will show up after we create a physical volume.
pvcreate /dev/sdb1 - creates the physical volume.
 Writing physical volume data to disk "/dev/sdb1"
  Physical volume "/dev/sdb1" successfully created
pvscan - will now show our new physical volume
 PV /dev/sda2   VG vg_redhat62     lvm2 [69.51 GiB / 0    free]
  PV /dev/sdb1                      lvm2 [320.00 GiB]
  Total: 2 [389.51 GiB] / in use: 1 [69.51 GiB] / in no VG: 1 [320.00 GiB]

STEP #4 - Create the Volume Group
 vgcreate vg_data /dev/sdb1 - This creates the Volume Group.  give it a name and point it to your physical disk.
  Volume group "vg_data" successfully created
You can run either vgscan or vgdisplay to see your new Volume Group.

STEP #5 - Create a Logical Volume
lvcreate -l 100%FREE -n lg_data vg_data - This creates the Logical Volume.
  Logical volume "lg_data" created
The "-l 100%FREE" mean to use 100% of available space and "-n" is for naming.  So I called the new Logical Volume "lg_data" and pointed it to my new Volume Group "vg_data".

To check on the new Logical Volume run either lvscan or lvdisplay.
lvscan
  ACTIVE            '/dev/vg_data/lg_data' [320.00 GiB] inherit
  ACTIVE            '/dev/vg_redhat62/lv_root' [33.14 GiB] inherit
  ACTIVE            '/dev/vg_redhat62/lv_home' [28.67 GiB] inherit
  ACTIVE            '/dev/vg_redhat62/lv_swap' [7.70 GiB] inherit

STEP #6 - Format the new Logical Volume
 mkfs.ext4 -m 0 /dev/vg_data/lg_data - File type is ext4 the "-m 0" don't reserve disk space for superuser. It save 5%
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
20971520 inodes, 83885056 blocks
0 blocks (0.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
2560 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 32 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

STEP #7 - Mount the Logical Volume
Create a directory that you want to mount the volume to.
mkdir data
mount /dev/vg_data/lg_data /data

These steps created a physical disk, a physical volume, a Volume group, a Logical volume, formatted the volume and mounted it.
 df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_redhat62-lv_root
                       33G  2.5G   29G   9% /
tmpfs                 2.8G  272K  2.8G   1% /dev/shm
/dev/sda1             485M   53M  407M  12% /boot
/dev/mapper/vg_redhat62-lv_home
                       29G  174M   27G   1% /home
/dev/mapper/vg_data-lg_data
                      315G  195M  299G   1% /data

As you can see are new LVM is listed last and is mounted.   

Thursday, April 26, 2012

Use secure keys to login automatically from windows



1. Using Putty login as the user you want to create the key.
2. type ssh-keygen –t rsa.


3. Now this will create two files id_rsa and id_rsa.pub under ~/.ssh
4. Now copy the id_rsa file (Private Key) to your document directory on your windows computer.
5. Use the Putty Key Generator program (right click on putty) that comes with Putty to import this file. (See step 6).



6. In the Putty Key Generator program click “Conversions”, then “import key” and choose the file id_rsa that  is in your document directory from step 4.
7. This is what it will look like.



8. Now copy the entire content of the big red box and paste it into a file in notepad, then save the file as “PuttyKey.
9. Copy the new file “PuttyKey” to the user’s home/.ssh directory on the Linux Server.
10. From the user’s home/.ssh directory on the Linux Server run this command.  “cat PuttyKey > authorized_keys”
11. Change permission on authorized_keys to 644.  “chmod 644 authorized_keys”










Now let’s get dl4term to use the Identity keys.


1. Open up dl4term.
2. Go to scripting and create a new script with these settings:











































3. Now click on SSH Options and browse to the id_rsa file that you saved on your computer in step 4











4. Give the script a name.  I called mine Login-SSH, now click “Save Script”.
5. Right click on the dl4term icon and choose properties.






6. Next to the last double quotes type in the name of the script you created.











































7. Now when you double click on dl4term it will automatically log you in.


-Jeff

Friday, February 24, 2012

Delete Network printer


I finally found out how to delete network printers in windows 7.  One way is to login as administrator’s privileges’.  That is too easy.

I wanted to be able to do it from the command prompt and logged in as a normal user.

Here is how you do it:

1      Open up a command prompt with elevated privileges.
2       From the command prompt enter this command and put the name of the printer you want to delete.

"rundll32 printui.dll PrintUIEntry /dn /n\\printerservername\printername"
remove the quotes