Python 3 - Finding Route53 record set values

This is a quick and simple script shows which are the route 53 record set values for a certain list of valid alias. It checks which Mysql servers on EC2 are being pointed from those alias.
The EC2 resources are filtered by tag system.
I use the powerful python module for creating the table to be shown.
Extra database info like replication lag using the heartbeat table and the master of the Mysql server is present in the outcome.

Python version 3.4

What to proof: How to use Texttable, Boto3

Python3 - Check Heartbeat value on Mysql on EC2 instances

This script searches for all auto scaling group that has the world "database" in their names, collects their Ip (private address only) and try a connection on to the Mysql server using a default configuration Mysql file. This is meant to work on different deployment and different environment, in my case a single environment can have multiple deployment id. The connection to the database is done by pymysql module and it does a query to the heartbeat table in the monitoring table. ( this is not configurable via parameters)

Python3 - Subprocess, Runs bash commands in a Python script

Subprocess for running Bash command on Shell

Subprocess module (newer) (After 3.5.3) The recommended approach to invoking subprocesses is to use the run() function for all use cases it can handle. (Current version 3.4.3) Uses .call

python-doc-subprocess

suprocess-examples

Python3 - Install & Variables

Python language is now highly demanded and being a good Python developer or basic at least is an important skill. Python 3 is being widely used and I made my mind up to get on with this version. As environment I use a Ubuntu 16.04

Thoughts about PMM Percona Monitoring and Management

PMM is the monitoring and management tools offers by Percona. It uses the power of hosted graphite to let you monitor metrics and Orchestrator to manage the replica chain and see what is the current status of the infrastructure. Percona-guide

MongoDB storage engine

It does affect

• How data is written to disk
• How data is deleted/removed from disk
• how data is read from disk
• Data structures used to store data

You can choice which Plug-in to use Actually

• MMAPv1
• WiredTiger

Aurora Auditing

Auditing-Aurora

You can use the high-performance Advanced Auditing feature in Amazon Aurora to audit database activity. To do so

• Enabling it is dynamic and it doesn't perform a cluster restart
• You can download the audit logs from console or aws cli
• You must be using Aurora 1.10.1 or greater to use Advanced Auditing.

Alert for Aurora failover

This Ruby script using the aws sdk library, check sometime if the Aurora cluster has failed over to another instance, and if so it raises a child Sensu alert.
It takes as parameter the db-cluster-id and it returns a exit code and exit message which can be interpreted by any monitoring system.
You can read and download it from my repo here

Github SSH URL configuration

This post is going to illustrate how you can easily configure and set up your github repository on Ubuntu

Sys DB for Mysql 5.6

This article is to sum up why the SYS DB for Mysql 5.6 is useful and what I think it is worth to use. You can install, and have a big picture what kind of tables/views looking at the Github repo

Aws Summit 2017

Aws Summit 2017 in London has brought out some interesting new features but not really new things for database industry.

I spoke with some famous Aws sponsors like DataDog, Skipjaq, and Rackspace. All of them were offering Cloud services and I learned that they have started to provide services hosting data on their clouds.

These are the companies that I got on with because their have good services, I’m not going to use them later on but it is worth to mention what their are trying to achieve.

Bringing in the cloud your applications and databases, you are losing control of where the data are physically

I believe some aspects of using the cloud is how you want to migrate to the cloud having a short period of downtime and how you overlook your statistics and the securities.

GPG encrypt tool

sdasdadadaGnuPG allows to encrypt and sign your data and communication, features a versatile key management system as well as access modules for all kinds of public key directories. GnuPG, also known as GPG, is a command line tool with features for easy integration with other applications.

DynamoDB Local and Global Secondary Indexes

Main features

• LSI allows another SORT key, SAME partition key
• Only at TABLE CREATION
• Max Five per table
• LSI are Asynchronously, reads from LSI are eventually consistency
• WCU and RCU Shared with the TABLE!!
• Eventual or Immediate Consistency

A LSI can make quicker and cheaper process quering only for the data that we need. For instance if we are insterested in a attribute that not always is available, we can have a LSI with the SORT KEY on this attribute.

Mysql Events 5.6

MySQL Events are tasks that run according to a schedule. Therefore, we sometimes refer to them as scheduled events. statements to be executed at one or more regular intervals Scheduled tasks of this type are also sometimes known as “temporal triggers”,

In MySQL, an event is uniquely identified by its name and the schema to which it is assigned.

DynamoDB notes

 

Introduction

• A fully managed, NoSQL Database service
• Predictable, fully managed perf with seamless scalability
• No visible servers
• Fully resilient and highly available
• Perf scales in a consistent, logical and linear way
• Full and deppe integration with AWS IAM, rich security model
• NoSQL database engin
• Shares characteristics with Key/Value adn Document Store DB's

Aurora–Out of Memory

 

Aurora cluster writer 2xlarge with 20GB memory.

I was doing bunch of updates on all orders, members, Catalogs table. The scope was to replace some information for certain fields.
The operation was quite heavy because the tables have more than 50 Milions rows and it had a condition with like and percent at the beginning.
It didn't generate the best execute plan because that condition doesn't use the index on that field.
I had to develop a store procedure that had in input the lower and upper id limits, the primary key, and the range that it needs to add at every run.
Quite simple procedure that allow to split up the updates in smaller pieces.

External slave slows down Aurora Master replication

As a heavy user of RDS Aurora cluster, I had some chances to test it and find a few problems.

I'm going to summarize and show in which case the Aurora can have a very bad behaviour in case it is in a chain of replica servers.