Browsed by
Category: software

ETF Leverage Simulator

ETF Leverage Simulator

The ETF Leverage Simulator synthesizes long-term leveraged ETF returns.

Leveraged ETF’s are exchanged traded funds that use derivatives and debt to magnify the returns of an underlying index. Such funds apply a leverage multiplier (typical 2x or 3x) to amplify an index’s actual returns on a daily basis.

This application allows the user to select an underlying index and then run a simulation to see how an ETF (applying daily leverage to the underlying index) would have performed over the same period.

For more details, see the application ETF Leverage Simulator (shiny.io), implementation ETF Leverage Simulator (github) or detailed analysis Long Term Leveraged ETF Performance.

Scaling MongoDB in EC2

Scaling MongoDB in EC2

Background

 

MongoDB has a nice guide for getting started with running mongoDB in EC2.

This article presents some additional enhancements for increasing performance while lower storage costs in EC2. A subsequent installment presents techniques for deploying replica sets and auto-scaling nodes.

AWS Storage Performance

 

Many sources, including the MongoDB guide linked above, recommend using EBS-backed volumes. “For production systems we recommend using EBS-optimized EC2 instances.” EBS backed volumes indeed provide the convenience of data persistence across instance reboots. However, since EBS volumes are network-based, they come with a performance penalty compared to EC2’s ephemeral-based (aka “instance”) storage.

EC2 Instance Storage by Instance Type

By default, EC2 instances (excluding smaller or specialized instance types) come with ephemeral storage. Size ranges from 160GB (for an m1.small instance), to 840GB (for m1.large), to several terabytes (for larger instances).

These ephemeral stores are associated with space reserved on physical storage attached to the underlying instances, made available at boot time and persistent for the life of the machine instance.

Because the instance space is accessible via the system bus on the host of virtual, rather than over the network for EBS volumes, disk performance is higher. This i/o performance is critical to the performance of NoSQL datastores as documented in MongoDB Write Performance.

For this reason, when using EBS volumes, the recommendation is made to utilize IOP (i/o provisioned) EBS volumes to ensure a specific levels of I/O performance on the networked EBS volume utilized by Mongo. IOPS (IO provisioned storage), however, brings additional costs both in the form of monthly per GB and per IOP fees.

Storage performance was measured using hdparm on three separate EC2 instances, one running MongoDB on a default EBS volume, one on a 200 IOPS provisioned EBS volume and one running against an instance store.

EBS default volume ~= 100 IOPS or ~35MB/s

% hdparm -t /dev/xvdf1
Timing buffered disk reads: 106 MB in 3.01 seconds = 35.26 MB/sec

EBS 200 IOPS ~= 87MB/s

% hdparm -t /dev/xvdf1
Timing buffered disk reads: 258 MB in 3.00 seconds = 85.98 MB/sec

EC2 Instance Store ~= 131MB/s or ~300 IOPS

% hdparm -t /dev/xvdg
Timing buffered disk reads: 394 MB in 3.01 seconds = 131.07 MB/sec

Default EBS volume performance is 100 IOPS. Default instance store performance is ~300 IOPS. EBS volume performance above the 100 IOPS level can be guaranteed by purchasing IOPS provisioned storage at additional cost.

AWS EBS IOPS vs. Instance Store Cost

 

Here’s a comparison of the cost of ephemeral vs. instance storage in EC2, for 400GB of underlying storage for use as a MongoDB store.

AWS/EC2 EBS and Provisioned IOPS monthly costs

The current rate for IOPS on EC2 is $0.125 per GB-month of provisioned storage plus $0.10 per provisioned IOPS-month. For a 400GB EBS volume with 200 IOP provisioned storage performance, this equates to $70, in storage costs, for every node in a replica set:

(400GB * $0.125 / GB / month + $0.10 * 200 IOPS) * replica set nodes = $70 / node

Cost and Performance

 

Combining the data above for both cost and performance, for 400GB of database disk storage:

MongoDB on EBS w/ 200 IOPS performance: $70/mo/node

MongoDB on Instance store w/ ~300 IOPS performance: $0/mo/node (via ephemeral storage)

Replica Sets

 

MongoDB supports several different mechanisms for replication, including master/slave, replica pairs and replica sets. As of MongoDB v1.8, the preferred mechanism is replica sets.

In the next installment of this article, I talk about EC2 deployment of MongoDB replica sets and how the cost benefits and high performance of instance storage can be realized while minimizing or mitigating the risk presented by ephemeral storage. I’ll also talk about how replica set nodes can be quickly deployed and initialized in order to achieve a level of auto-scaling and fault tolerance in a production environment, with nodes deployed in sufficient number and across EC2 availability zones to ensure high availability.

 

Bengaluru, India

Bengaluru, India

Taj West End

I had the privilege of visiting Bangalore for two weeks. When I wasn’t working with my colleagues at their office in Manyata Park, had the opportunity to explore the city and make a day trip to Mysore.

Manyata Tech Park

Bangalore is the capital of Karnataka and commercial hub of the Deccan (Karnataka and Andhra Pradesh states). According to National Geographic, the city is the “world’s third most important IT city… a more international city than even Mumbai.”

Bangalore is indeed an amazing place; a combination of high-tech and old world. It is the third largest city in India and the fastest growing city in Asia, increasing in population from just over 1 million people in 1970 to nearly 8 million today.

Commercial Street, Bangalore

One of the most incredible things about Bangalore is the population density. Bangalore has ~8k people per square kilometer, or about 4 times that of New York City. Mumbai weighs in at over 20k/km^2, roughly ten times more people per area than in the most dense American cities. Density exceeds even Hong Kong by a factor of 3.

Lal Bagh Rock

Formerly known as the “Garden City”, a few signs of Bangalore’s greener past still remain. One is Lal Bagh, a 240 acre garden near the city center which has hundreds of species of plants and many massive ancient trees. The park also features Kempegowda tower, built on the surface of Lal Bagh Rock, one of the oldest rock formations on earth, dating from 3 billion years ago.

Bangalore to Mysore

Approximately 150km outside of Bangalore is the city of Mysore. It served as the capital of the region through the reign of the Wodeyar dynasty and until 1947 when administrative power was shifted to Bangalore.

Ambivalas Palace at Mysore

The Ambivalas Palace built by the ruling Wodeyars is one of the most popular tourist attractions in India.

“The architectural style of the palace is commonly described as Indo-Saracenic, and blends together Hindu, Muslim, Rajput, and Gothic styles of architecture. It is a three-storied stone structure, with marble domes and a 145 ft five-storied tower.” The palace contains a wooden ‘howda’ (elephant sadle) decorated with 84kg(!) of gold.

Mysore to Chamundi Hill

The palace also contains many images of Durga (aka Chamundeshwari), the goddess, who according to Hindu mythology, killed the demon Mahishasura, allowing good to triumph over evil. The Chamundi temple which sits atop nearby hills on the outskirts of the city was built in honor of the goddess.

Chamundi Temple

The road to Mysore from Bangalore passes through the towns of Channaputna, known for its wooden crafts and toys, Maddur and Mandya. It also cuts across numerous sugar cane fields from which “jaggery” is extracted.

Jaggery Factory

Jaggery is made by boiling raw sugarcane juice in large shallow vessels. Jaggery is used in a variety of sweet dishes in India; it can also be added to curries. Jaggery is considered to be healthier than other sweeteners because it is prepared without the use of chemicals and it contains minerals not found in sugar. It has also been found to prevent lung damage from “particulate matter such as coal and silica dust”.. potentially quite useful when spending any time riding an auto-rickshaw on the streets of central Bangalore.

Bullock Cart Sharing the Road

Speaking of, one of my colleagues treated me to a ride on the back of his motorbike through Bangalore’s notorious traffic where it isn’t uncommon to see three cars side-by-side, jockeying for two lanes along side bull carts, pedestrians, autorickshaws and the occasional wayward cow or stray dog.

The video can be found here: Bangalore Motorbike Ride

The Law

If you are watching the video thinking “well, that isn’t nearly as cool as the Evel-Knievel vid I just watched yesterday”… Please! The driver was going easy on me. Thankfully, as I was holding the iphone in one hand, clutching an umbrella and the edge of the seat in the other, all the while trying not to lose my Nikon and the lense collection dangling from my shoulder. For some reason, I thought I’d be able to shoot some stills and video. Maybe not.

On the Cell

One of the mysteries of Bangalore is why they need so many traffic cops (literally dozens of them at some intersections) to enforce so few traffic laws. Driving is absolute and total chaos. Out of the chaos, though, some semblance of order seems to emerge from the honking, flashing lights and arm waving.

Brigade Road

I’m not talking about the occasional honk or flash. I’m talking about continuous honking and sometimes frantic dimming of the lights. I initially thought it might be morse code. Meanwhile, everyone seems to maintain perfect composure.

One Lane, Many Vehicles

When I say hand gestures, I’m talking about waves of the arm to allow or warn of a merge. I’m sure there’s a lesson in there somewhere, if not for Boston drivers, perhaps for a research paper on chaos theory or self-organizing systems.

Read More Read More

AOD255e upgrade

AOD255e upgrade

AOD255e

Acer Aspire AOD255e, Unwrapped
AOD255e Unwrapped

The new Acer Aspire One AOD255E features intel’s new dual-core Atom n570.

The n570 is the first n-series Atom chip to include hardware virtualization (or VT-x) extensions. Add dual cores with hyperthreading, and the system has the potential to perform more like a higher end notebook.

Not bad for $329, but the system ships with only 1GB of RAM and a 250GB mechanical HD.

SSD and RAM Upgrade

To bring these components up to par with the processor, increase the performance and reliability, I opted for a 128GB SSD and 2GB DDR3 memory module.

AOD255e Keyboard
AOD255e Keyboard

Although AOD255e isn’t designed to be user serviceable, replacing the HD and RAM isn’t too difficult.

  • Power off the machine and remove the battery
  • Locate the tabs along the upper edge of the keyboard above the F1, F4, F8, F12 and DEL keys.
  • Using a flat-head screwdriver and working from right to left, starting above the DEL key, carefully depress each tab and pry up the edge of the keyboard.
  • Once all of the tabs are depressed and the upper edge of the keyboard released, it can be carefully removed.
  • Remove all the screws marked “1”
  • Push a blunt object through the access hole marked, releasing the access panel on the bottom of the system.
  • Locate and replace the desired components

Fedora

Finally, I replaced Windows 7 with Fedora 14. Yes, it is indeed twice as good, maybe more… :/

System runs very smoothly with KDE4 and all visual effects enabled.

AOD255e Running Fedora 14
AOD255e Running Fedora 14

Here’s /proc/cpuinfo showing dual cores with 2 hyperthreaded execution units each and ‘vmx’ flag indicating VT-x support.


processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 28
model name : Intel(R) Atom(TM) CPU N570 @ 1.66GHz
stepping : 10
cpu MHz : 1000.000
cache size : 512 KB
physical id : 0
siblings : 4
core id : 1
cpu cores : 2
apicid : 3
initial apicid : 3
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm movbe lahf_lm tpr_shadow vnmi flexpriority
bogomips : 3325.21
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:

To perform the upgrade to Fedora, I used ‘BFO’, a small network bootloader.

The BFO boot image is only 300KB, can be written to a USB stick and used to do a network-based bootstrap of the Fedora installation process. Find Fedora’s BFO image here…
http://boot.fedoraproject.org

Summary

$329 – Acer Aspire One AOD255E-1664 Netbook, w/ dual-core Atom n570 and 10.1″ LED
$24 – Crucial 2GB 204-PIN PC3-8500 SODIMM DDR3 Memory Module
$240 – Kingston SSDNow V Series 128 GB SATA 3 GB/s $240 2.5″ Kingston 128GB SATA3 2.5″ SSD

jpcap

jpcap

jpcap capture console

Summary

jpcap is a set of Java classes which provide an interface and system for network packet capture. A protocol library and tool for visualizing network traffic is included.

jpcap hides the low-level details of network packet capture by abstracting many network packet types and protocols into Java classes. Internally, jpcap implements bindings to the libpcap system library through JNI (the Java Native Interface).

jpcap utilizes libpcap, a widely deployed shared-library for capturing user-level packets. libpcap must be installed on your system in order to use jpcap.

jpcap packet visualization

jpcap consists of a small shared-library which wraps libpcap plus a collection of Java classes. The shared-library component provides event hooks, communication and data conversion between a running Java VM and libpcap. The ‘capture’ package contains the core capture system. The ‘net’ package contains abstractions for many network packet types
and protocols. The ‘simulator’ package contains a network simulator.

Download

jpcap was originally developed in 2001, hosted at sourceforge.net and now available at github. The project is the work of more than a dozen contributors.

 

JChalkIR

JChalkIR

net.ultrametrics.palm.JChalkIR is a java ‘Spotlet’ for the experimental KJava virtual machine for handheld devices. Allows two or more people to collaborate on a simple sketch.

 

note: Application requires Sun’s KVM (Early Access version 0.1 [DR4] or newer). The SDDK and VM can be found on Sun’s JDC KVM page. If you are still using the Java One KVM [DR3 or older], try the old JChalkIR. Source and javadoc jar files provided are compatible with the jar tool packaged with jdk1.2 and later.