Thursday, November 30, 2017

"The disk is write protected" error on a Windows Failover Cluster Node VM

Lately, I came across this issue where database team was unable to start the sql service on SQL cluster nodes and when they checked SQL logs, found the drive where temp database stored was no more writable (I have seen this issue a few times in past and if I remember correctly every time this happened on Windows Server 2008 R2 server failover cluster nodes) .

When tried to create a new object in this drive, found no option to do so and checking the disk for errors from disk properties => Tools, end up with an error like,


I am not sure what caused this issue however, found following Volume Shadow Copy service error entries in event viewer, “A critical dynamic disk is a virtual hard disk. This is an unsupported configuration.” So, suspect this has something to do with volume shadow copy and Microsoft server cluster.


We can fix this Write protected disk issue by clearing the read only attribute from volume level (in my case clearing the read only attribute from disk didn’t work so suggest you clear the attribute at lowest level).

To do so follow these steps:

First open disk management and note down the disk and volume name/number of affected drive/volume.

  1. Open a command prompt, type in Diskpart and then press Enter.
  2. Run the command “List Volume” and press Enter.
  3. Now decide the Volume name, dive letter (Affected Read only volume, identified earlier from Disk Management).
  4. Select the affected volume by using “Select volume x” command, where x is the volume number
  5. One the affected volume is selected, now to clear the “Read-only” attribute, run the command “attributes volume clear readonly”. 
                       
And with this you are done, now your disk/volume should be writable.

You can check the same by running “detail volume” or “Attributes volume” command.

Note: Here one may ask here why I didn’t verify the disk/volume read/write attribute earlier and the short answer is yes, I did check and interestingly readonly attribute was set to No but still I have to clear the readonly attribute to make it writable again 😉.

In Case if your folders inside the drive are still not writable then run following cmd to remove the Read-only attribute and to set the System attribute.

C:\attrib -r +s drive:\folder_name 

Hope this would be helpful to others.........That's it :)


Sunday, October 1, 2017

HP SPP upgrade failing with unexpected errors

This week while upgrading server Firmware/Drivers on HP ProLiant G9 servers using latest HP SPP, on one server SPP online deployment return with following,


And when we checked install logs found all updates failed. As lately I had faced an issue where HPSUM inventory was failing due to missing HP management tools and as per my past experience I was aware about the fact, VMware update installations or individual vib install may fail if server has no esxi image profile attached/image profile got corrupted hence thought of cross checking the same.

On checking found this server had no host image profile defined,


When checked the esxupdate.log file found entries of the name of HPE custom esxi 6.0 u2 install file that mean the install was done using the correct ISO image and somehow it got corrupted later.

Then checked for the available image profiles by searching for Imgdb.tgz file (this is the image profile backup and by looking at its size we can get an idea if the image profile got corrupted), this will list the two Imgdb.tgz files as follows,


By looking at the size of these files, size of one imgdb.tgz file is comparatively very less and seems got corrupted.

We can fix this broken image profile issue by either doing an upgrade or fresh install. There is one more way to fix the above issue and that is by copying/replacing the host image profile of this host with an image profile from a working host.

We can do that as follows,

Use winscp or any other ftp client to connect to any healthy Esxi host and browse to /bootbank dir and copy the Imgdb.tgz file to your system and now connect to the host that is missing the image profile and paste the earlier copied Imgdb.tgz file to /tmp dir.

Now

Remove the corrupted imgdb.tar file,

rm /bootbank/Imgdb.tgz

Extract the Imgdb.tgz dir copied from healthy host,

tar -xzf /tmp/Imgdb.tgz

then copy the working image profile and related vibs to /var/db/esximg/profiles and /var/db/esximg/vibs directories,

cp /tmp/var/db/esximg/profiles/* /var/db/esximg/profiles/
cp /tmp/var/db/esximg/vibs/* /var/db/esximg/vibs/

After this run following cmd to create backup of image,

/sbin/auto-backup.sh

Now when you check for the installed profile, it will show you one. It wouldn’t reflect the host profile name in host summary until you re-start the host mgmt agents or reboot the host.

After doing above, when I re-initiated the SPP online update, this time host firmware/driver got updated without any further issue.


That’s it… J


HP SPP online update: HPSUM host inventory failed with an error like re-enter the credentials

Lately, I was upgrading HP servers firmware/drivers using HP SPP online upgrade method and for one HP ProLiant server after adding the node to HP SUM inventory when I try to inventory my box, got an inventory error and it asks me to re-enter my credentials while the entered credentials were correct.
Usually when a node is added you must inventory it against the SPP repository so HPSUM could compare and determine what is needed for your box.
When googled about the issue, most of the results pointed out this host seems missing some HPE VIBs that must be on the box prior to attempting to use HP SUM, this was strange as this host was installed using the HPE provided custom ESXi 6.0 U2 ISO image.

Here in order to fix the issue, downloaded and installed the HPE custom ESXi 6.0 U2 image bundle and after that there was no issue with SPP update (as this also installed all missing HP management components).

To upgrade the esxi install using VMware offline update bundle zip use,

 esxcli software vib update -d /vmfs/volumes/…./vmware_update_offline_bundle.zip

There are other ways too to upgrade the esxi host, like installing one of the available profiles in offline update bundle.

To list the installed image profile we can use following cmd,

esxcli software profile get

Now list the available image profile in download VMware update offline bundle,

 esxcli software profile list -d /vmfs/volumes/…./vmware_update_offline_bundle.zip

The output of above cmd would list the available image profiles from the downloaded vmware_update_offline_bundle.zip, note down the name of image profile you want to install, now

 esxcli software profile install -d /vmfs/volumes/…./vmware_update_offline_bundle.zip -p vmware_inage-profile name

This will install/update the desired esxi image profile.

That’s it… J


Thursday, August 31, 2017

AWS Certified SysOps Administrator - Associate exam experience and few exam related tips

I sat the AWS Certified SysOps Administrator - Associate exam today and passed, in this post I would share my exam experience and a few exam related tips.


There were 55 questions in this exam and and most of the questions were quite trickier than AWS Solution Architect - Associate exam and I hardly found any definitive question however that doesn't mean you would not read and understand definitions of different services ;) 

There were scenario based questions about changing tenancy of a running instance, placement group, EBS snapshots, Instance/ephemeral storage, ELB, Auto-scaling, Route53 records, Routing policy, IAM policies, S3 storage related backup scenario, network connectivity related questions in VPC, Cloudwatch alarms, health checks, Multi-AZ RDS & Read replica, Elastic beanstalk, Cloud Formation, Opswork and AWS shared responsibility model etc.

Here is my exam score, Overall Score: 83%

Topic Level Scoring:
1.0 Monitoring and Metrics: 87%
2.0 High Availability: 100%
3.0 Analysis: 55%
4.0 Deployment and Provisioning: 87%
5.0 Data Management: 85%
6.0 Security: 75%
7.0 Networking: 100%

About the preparation of this exam, learn ins and out of all the AWS services covered in this exam by heart and the best option to do that is, watch this certification's training CBTs few times and once you are done with that now review the FAQ of these services and read AWS security withe paper once. 

I used Ryan Kroonenburg aka AcloudGuru's ( AWS Certified SysOps Administrator - Associate  online ILVT course and Elias Khnaser's AWS Certified SysOps Administrator - Associate course on Pluralsight however i must say any one of these course is suffice to pass the exam.

I would also suggest going for AWS solution architect course/exam first, as it would give you a better understanding about most of the AWS services and provide a solid base for this SysOps exam.

My Previous post about AWS Certified Solution Architect exam experience and few exam related tips can be found here.

Finally before going for the exam, make sure you also practiced on various sample questions available over web, however don't depend on then them as you would find many wrong answers there (however you will be able to to catch them only if you know the ins and out of different serveries by heart).

Hope this would be helpful in passing you AWS Sysops exam....Best of Luck!

That's it.... :)


Thursday, August 17, 2017

Windows Server reboot/shutdown events in event viewer

Ever found yourself wondering about an unexpected system reboot, these event IDs are very useful while one is investigating the cause of unexpected system shutdown/reboot. 
I not sure about others but but always found hard remembering these event IDs so making a note for future reference and believe others will also find it useful.

Server reboot/shutdown events:
  •  Event ID 6005: “The event log service was started.” This is synonymous to system startup.
  •   Event ID 6006: “The event log service was stopped.” This is synonymous to system shutdown.
  •   Event ID 6008: "The previous system shutdown was unexpected." Records that the system started  after it was not shut down properly.
  •   Event ID 6009: Indicates the Windows product name, version, build number, service pack number,   and operating system type detected at boot time.
  •    Event ID 6013: Displays the uptime of the computer. There is no TechNet page for this id.
  •    Event ID 1074: "The process X has initiated the restart / shutdown of computer on behalf of user     Y for the following reason: Z." Indicates that an application or a user initiated a restart or                  shutdown.
  •    Event ID 1076: "The reason supplied by user X for the last unexpected shutdown of this computer    is: Y." Records when the first user with shutdown privileges logs on to the computer after an           unexpected restart or shutdown and supplies a reason for the occurrence.
Note: In case of unexpected shoutdown due to power failure, there would be no event created.

To know the system boot Time:
  C:\systeminfo | find /i “boot time”


That's it... :)