PhD student has to write blog posts regularly, here you go.

"PhD is a career like running your own business". I swung by an event for new incoming students to UChicago and herd such a sentence. I set my heart on learning a couple more things from my fellow students on how I should live my PhD life. I read a couple of pages in a book on how to write academic papers and one advice was that you should call your supervisor: advisor. It was repeated in the talk. It seems to me that in this matter the nomenclature is really of importance. It's not only your advisor who can teach you new aspects but also other students around you, of course. The only requirement is that the other people should be interested in a similar topic like you. The database group at UChicago is growing rapidly and we hope that our new chair will be willing to help with this endeavor. It would be too cumbersome to create a story from the thoughts that I heard today so let me just enumerate them and comment a little bit on each of them.

1. You get what people think you should get. This is very true. We have to master the skill of communication, especially about what we work on, what is our research about and what is in for you. Giving a talk is always a challenge for me, but I love it. I'm always thrilled and sometimes even a little bit nervous, however, once I am there on the stage, know that my material is well-prepared and start communicating with the audience, then there is no better time in my life, seriously.
2. Don't do something off truck. So, communicate with your advisor frequently and make sure that you both agree on the next steps. I don't want to make my advisor being frustrated, sure thing. This is more about how you want to run your business, what your vision is for an excellent new approach/technique/method/algorithm/theory/experiments. Just make sure that your supervisor agreed with you and remember to calibrate your research trajectory.
3. Spend some time improving yourself. I know that I must work more on math & English. This is math that helps me to keep sane in this unstructured world of words/thoughts/opinions and get on the right truck with my own thinking. At the same time, you mush speak/read/write in English and do it perfectly, otherwise you loose a lot being in the US or other English speaking countries.
4. Write a technical blog. I'll write more technical stuff, I promise. It'll be on compression, Spark, parquet (and other data formats), data migration, some other databases (MonetDB), and I will comment on some papers.
5. This blog has to be keyword search-able, I'm glad it is.
6. Revise your blog posts from time to time.
7. Endurance and self-motivation.
8. Take ownership of your life and your research.
9. Manage yourself as you were a superstar.
10. Act thoughtfully.
11. Act intelligently.
12. Act as a leader.

Let it be all for now.

San Francisco from another perspective

If you are going to San Francisco, take some smile with you. This city just requires you to admire the beautiful and sunny days. This vibrant place let you experience an amazing mixture of tastes from posh Financial District to sad places brim with homeless people. You can do a traditional city-oriented sight-seeing or go to special places, such as, California Academy of Sciences or the Young Museum in the Golden Gate Park. The crucial point is this, it's all about people. Ask them what the best place to visit is and you'll get a basket of tips and tricks :)

Transamerica Pyramid - tall, landmark building shaped like a narrow pyramid (view from Hyatt Regency in the Financial District).

The Sun penetrating the Financial District.

Railways of the cable car and steep hill in the background.

Manually operated cable car (the icon of the city and the last such relic in the world).

This is not the Golden Gate bridge but the San Francisco - Oakland Bay bridge. It also looks marvelous.

A homeless person at one of the Bart stations.

Gorgeous butterfly in the Golden Gate Park (California Academy of Sciences).

Check your paper for IEEE compliance through the IEEE PDF eXpress site

I found that the easiest way to make your pdf compliant with the IEEE requirements is to just print your pdf to another pdf! This works like a charm. I had a few problems, which are enumerated in the table below. I tried removing the urls, I even tried these hacks: http://mohamednabeel.blogspot.com/2009/10/fixing-font-not-embedded-issue-to-pass.html. However, the only thing that you have to do is to grab the document generated by latex, go to File->Print...->Print to file (choose another location/name) and voila, it works! :)

	If the report says:	Possible cause(s)	Solution
(1)	Bookmarks found in document	(1) User inserted bookmarks into the PDF.	Click
(2)	Document contains security	(2a) User applied security to some or all elements of the PDF	Click
		(2b) If source is Microsoft Word— PDFMaker is set to apply security to all PDFs	Click
(3)	Font ### is not embedded	(3a) The font file does not exist on the system that created the PDF	Click
		(3b) The font is not being found by Distiller	Click
		(3c) The font is not embeddable	Click
		(3d) Using Office 2003 or 2007	Click
(4)	Font ### is not subsetted	(4a) PDF conversion options are not set correctly	Click
		(4b) Headers/footers were added using Acrobat 6 function (or other application)	Click
		(4c) Using Office 2003 or 2007	Click
(5)	Document contains link annotations	(5) User has applied a link to text in the document.	Click
(6)	Document contains form fields	(6) User has used form fields in the document.	Click

Thought of the day

"If you want to build a ship, don’t drum up the men to gather wood, divide the work, and give orders. Instead, teach them to yearn for the vast and endless sea." Antoine de Saint-Exupéry

Another RUBY lesson

adam@gaia:~$ ruby hello.rb
hello.rb:2:in `<main>': undefined method `wait' for Thread:Class (NoMethodError)
adam@gaia:~$ ruby hello.rb
In another thread
adam@gaia:~$ ruby hello.rb
In thread 0In thread 5In thread 1In thread 9
In thread 4In thread 2In thread 8In thread 3
In thread 7In thread 6

adam@gaia:~$ ruby hello.rb
In thread 0
In thread 3
In thread 4
In thread 6
In thread 1
In thread 2
In thread 9
In thread 5
In thread 7
In thread 8
adam@gaia:~$ irb -r ./hello.rb
irb(main):001:0> pt_1
NameError: undefined local variable or method `pt_1' for main:Object
    from (irb):1
    from /usr/bin/irb:12:in `<main>'
irb(main):002:0> load('hello.rb')
=> true
irb(main):003:0> pt_1
NameError: undefined local variable or method `pt_1' for main:Object
    from (irb):3
    from /usr/bin/irb:12:in `<main>'
irb(main):004:0> class Point
irb(main):005:1>   attr_accessor :x, :y
irb(main):006:1> end
=> nil
irb(main):007:0>
irb(main):008:0* class Line
irb(main):009:1>   attr_accessor :pt_1, :pt_2
irb(main):010:1> end
=> nil
irb(main):011:0>
irb(main):012:0* pt_1 = Point.new
=> #<Point:0x000000014ea100>
irb(main):013:0> pt_1.x = 10
=> 10
irb(main):014:0> pt_1.y = 20
=> 20
irb(main):015:0>
irb(main):016:0* pt_2 = Point.new
=> #<Point:0x000000014a38e0>
irb(main):017:0> pt_2 = 50
=> 50
irb(main):018:0> pt_2 = 100
=> 100
irb(main):019:0>
irb(main):020:0* line = Line.new
=> #<Line:0x00000001496550>
irb(main):021:0> line.pt_1 = pt_1
=> #<Point:0x000000014ea100 @x=10, @y=20>
irb(main):022:0> line.pt_2 = pt_2
=> 100
irb(main):023:0> pt_1
=> #<Point:0x000000014ea100 @x=10, @y=20>
irb(main):024:0> line
=> #<Line:0x00000001496550 @pt_1=#<Point:0x000000014ea100 @x=10, @y=20>, @pt_2=100>
irb(main):025:0> line.pt_1
=> #<Point:0x000000014ea100 @x=10, @y=20>
irb(main):026:0> line.pt_1.x
=> 10
irb(main):027:0> line.pt_1.y
=> 20
irb(main):028:0> line.pt_2.c
NoMethodError: undefined method `c' for 100:Fixnum
    from (irb):28
    from /usr/bin/irb:12:in `<main>'
irb(main):029:0> line.pt_2.x
NoMethodError: undefined method `x' for 100:Fixnum
    from (irb):29
    from /usr/bin/irb:12:in `<main>'
irb(main):030:0> line.pt_1.y
=> 20
irb(main):031:0> class Point
irb(main):032:1>   attr_accessor :x
irb(main):033:1>
irb(main):034:1*   def y
irb(main):035:2>     x + 10
irb(main):036:2>   end
irb(main):037:1> end
=> nil
irb(main):038:0>
irb(main):039:0* class Line
irb(main):040:1>   attr_accessor :pt_1, :pt_2
irb(main):041:1> end
=> nil
irb(main):042:0>
irb(main):043:0* pt_1 = Point.new
=> #<Point:0x0000000123e000>
irb(main):044:0> pt_1.x = 10
=> 10
irb(main):045:0> pt_1.y = 20
=> 20
irb(main):046:0>
irb(main):047:0* pt_2 = Point.new
=> #<Point:0x0000000122cee0>
irb(main):048:0> pt_2 = 50
=> 50
irb(main):049:0> pt_2 = 100
=> 100
irb(main):050:0>
irb(main):051:0* line = Line.new
=> #<Line:0x000000017189e0>
irb(main):052:0> line.pt_1 = pt_1
=> #<Point:0x0000000123e000 @x=10, @y=20>
irb(main):053:0> line.pt_2 = pt_2
=> 100
irb(main):054:0> line.pt_1.y
=> 20
irb(main):055:0>
adam@gaia:~$ irb -r ./hello.rb
irb(main):001:0> Point.new.x
ArgumentError: wrong number of arguments (0 for 2)
    from /home/adam/hello.rb:2:in `initialize'
    from (irb):1:in `new'
    from (irb):1
    from /usr/bin/irb:12:in `<main>'
irb(main):002:0> Point.new(10, 20).x
=> 10
irb(main):003:0> Point.new(10, 20).y
=> 20
irb(main):004:0> Point.new(10, 20).adam
Called adam but I only respond to [:method_missing, :x, :x=, :y, :y=]
=> nil
irb(main):005:0> load('hello.rb')
=> true
irb(main):006:0> Point.new(10, 20).x
=> 10
irb(main):007:0> Point.new(10, 20).y
=> 20
irb(main):008:0> Point.new(10, 20).adam
Called adam but I only respond to [:!, :!=, :!~, :<=>, :==, :===, :=~, :__id__, :__send__, :class, :clone, :define_singleton_method, :display, :dup, :enum_for, :eql?, :equal?, :extend, :freeze, :frozen?, :hash, :initialize_clone, :initialize_dup, :inspect, :instance_eval, :instance_exec, :instance_of?, :instance_variable_defined?, :instance_variable_get, :instance_variable_set, :instance_variables, :is_a?, :kind_of?, :method, :method_missing, :methods, :nil?, :object_id, :private_methods, :protected_methods, :public_method, :public_methods, :public_send, :respond_to?, :respond_to_missing?, :send, :singleton_class, :singleton_methods, :taint, :tainted?, :tap, :to_enum, :to_s, :trust, :untaint, :untrust, :untrusted?, :x, :x=, :y, :y=]
=> nil
irb(main):009:0> str = "Adam 6\nBrian 10\nCarol 13\nBrian 14\nAdam 23\nCarol 14"
=> "Adam 6\nBrian 10\nCarol 13\nBrian 14\nAdam 23\nCarol 14"
irb(main):010:0> puts str
Adam 6
Brian 10
Carol 13
Brian 14
Adam 23
Carol 14
=> nil
irb(main):011:0> str.lines
=> #<Enumerator: "Adam 6\nBrian 10\nCarol 13\nBrian 14\nAdam 23\nCarol 14":lines>
irb(main):012:0> str.lines.to_a
=> ["Adam 6\n", "Brian 10\n", "Carol 13\n", "Brian 14\n", "Adam 23\n", "Carol 14"]
irb(main):013:0> str.lines.map { |line| line.split }
=> [["Adam", "6"], ["Brian", "10"], ["Carol", "13"], ["Brian", "14"], ["Adam", "23"], ["Carol", "14"]]
irb(main):014:0> str.lines.map { |line| line.split }.map { |name, score| [name, score.to_i] }
=> [["Adam", 6], ["Brian", 10], ["Carol", 13], ["Brian", 14], ["Adam", 23], ["Carol", 14]]
irb(main):015:0> str.lines.map { |line| line.split }.map { |name, score| [name, score.to_i] }.group_by { |name, score| name }
=> {"Adam"=>[["Adam", 6], ["Adam", 23]], "Brian"=>[["Brian", 10], ["Brian", 14]], "Carol"=>[["Carol", 13], ["Carol", 14]]}
irb(main):016:0> str.lines.map { |line| line.split }.map { |name, score| [name, score.to_i] }.group_by { |name, score| score }
=> {6=>[["Adam", 6]], 10=>[["Brian", 10]], 13=>[["Carol", 13]], 14=>[["Brian", 14], ["Carol", 14]], 23=>[["Adam", 23]]}
irb(main):017:0> str.lines.map { |line| line.split }.map { |name, score| [name, score.to_i] }.group_by { |name, score| name[0] }
=> {"A"=>[["Adam", 6], ["Adam", 23]], "B"=>[["Brian", 10], ["Brian", 14]], "C"=>[["Carol", 13], ["Carol", 14]]}
irb(main):018:0> str.lines.map { |line| line.split }.map { |name, score| [name, score.to_i] }.group_by { |name, score| name }
=> {"Adam"=>[["Adam", 6], ["Adam", 23]], "Brian"=>[["Brian", 10], ["Brian", 14]], "Carol"=>[["Carol", 13], ["Carol", 14]]}
irb(main):019:0> str.lines.map { |line| line.split }.map { |name, score| [name, score.to_i] }.group_by { |name, score| name }.map { |name, name_and_scores| name_and_scores }
=> [[["Adam", 6], ["Adam", 23]], [["Brian", 10], ["Brian", 14]], [["Carol", 13], ["Carol", 14]]]
irb(main):020:0> str.lines.map { |line| line.split }.map { |name, score| [name, score.to_i] }.group_by { |name, score| name }.map { |name, name_and_scores| name_and_scores.map(&:last) }
=> [[6, 23], [10, 14], [13, 14]]
irb(main):021:0> str.lines.map(&:split).map { |name, score| [name, score.to_i] }.group_by(&:first).map { |name, name_and_scores| name_and_scores.map(&:last) }
=> [[6, 23], [10, 14], [13, 14]]
irb(main):022:0> str.lines.map(&:split).map { |name, score| [name, score.to_i] }.group_by(&:first).map { |name, name_and_scores| name_and_scores.map(&:last).reduce(:+) }
=> [29, 24, 27]
irb(main):023:0> str.lines.map(&:split).map { |name, score| [name, score.to_i] }.group_by(&:first).map { |name, name_and_scores| [name_and_scores.map(&:last).reduce(:+)] }
=> [[29], [24], [27]]
irb(main):024:0> str.lines.map(&:split).map { |name, score| [name, score.to_i] }.group_by(&:first).map { |name, name_and_scores| [name, name_and_scores.map(&:last).reduce(:+)] }
=> [["Adam", 29], ["Brian", 24], ["Carol", 27]]
irb(main):025:0> str.lines.map(&:split).map { |name, score| [name, score.to_i] }.group_by(&:first).map { |name, name_and_scores| [name, name_and_scores.map(&:last).reduce(:+)] }.to_h
NoMethodError: undefined method `to_h' for [["Adam", 29], ["Brian", 24], ["Carol", 27]]:Array
    from (irb):25
    from /usr/bin/irb:12:in `<main>'
irb(main):026:0> str.lines.map(&:split).map { |name, score| [name, score.to_i] }.group_by(&:first).map { |name, name_and_scores| [name, name_and_scores.map(&:last).reduce(:+)] }.to_hash
NoMethodError: undefined method `to_hash' for [["Adam", 29], ["Brian", 24], ["Carol", 27]]:Array
    from (irb):26
    from /usr/bin/irb:12:in `<main>'
irb(main):027:0> RUBY_VERSION
=> "1.9.3"
irb(main):028:0> Hash[str.lines.map(&:split).map { |name, score| [name, score.to_i] }.group_by(&:first).map { |name, name_and_scores| [name, name_and_scores.map(&:last).reduce(:+)] }]
=> {"Adam"=>29, "Brian"=>24, "Carol"=>27}
irb(main):029:0> str
=> "Adam 6\nBrian 10\nCarol 13\nBrian 14\nAdam 23\nCarol 14"
irb(main):030:0> str
=> "Adam 6\nBrian 10\nCarol 13\nBrian 14\nAdam 23\nCarol 14"
irb(main):031:0> Hash[str.lines.map(&:split).map { |name, score| [name, score.to_i] }.group_by(&:first).map { |name, name_and_scores| [name, name_and_scores.map(&:last).reduce(:+)] }]
=> {"Adam"=>29, "Brian"=>24, "Carol"=>27}
irb(main):032:0> Hash[str.lines.map(&:split).map { |name, score| [name, score.to_i] }.group_by(&:first).map { |name, name_and_scores| [name, name_and_scores.map(&:last).reduce(:+)] }].max_by(&:last)
\=> ["Adam", 29]
irb(main):033:0> Hash[str.lines.map(&:split).map { |name, score| [name, score.to_i] }.group_by(&:first).map { |name, name_and_scores| [name, name_and_scores.map(&:last).reduce(:+)] }.sort_by(&:last)]
=> {"Brian"=>24, "Carol"=>27, "Adam"=>29}
irb(main):034:0> Hash[str.lines.map(&:split).map { |name, score| [name, score.to_i] }.group_by(&:first).map { |name, name_and_scores| [name, name_and_scores.map(&:last).reduce(:+)] }.sort_by(&:last).reverse]
=> {"Adam"=>29, "Carol"=>27, "Brian"=>24}
irb(main):035:0> Hash[str.lines.map(&:split).map { |name, score| [name, score.to_i] }.group_by(&:first).map { |name, name_and_scores| [name, name_and_scores.map(&:last).reduce(:+)] }.sort_by(&:last).reverse.lazy]
NoMethodError: undefined method `lazy' for [["Adam", 29], ["Carol", 27], ["Brian", 24]]:Array
    from (irb):35
    from /usr/bin/irb:12:in `<main>'
irb(main):036:0> Hash[str.lines.map(&:split).map { |name, score| [name, score.to_i] }.group_by(&:first).map { |name, name_and_scores| [name, name_and_scores.map(&:last).reduce(:+)] }.sort_by(&:last).reverse]
=> {"Adam"=>29, "Carol"=>27, "Brian"=>24}
irb(main):037:0>

RAID

I always forget what each RAID looks like. Here, I don't want to repeat the standard information but give some comparison and consequences of using each RAID. RAID 2 and 3 are not used any more, I suppose that the same holds for RAID 4. RAID 5 is a suitable solution for databases.

The main difference between RAID 4 and RAID 5 is that in RAID 4 you can have many reads at a time (if you don't check the parity for every read), but you cannot have many writes at a time because each write have to update the parity disk. The RAID 5 supports both many reads and many writes at the same time (providing the writes do not want to update the parity on the same disk). All the reads and writes for RAID 2 and RAID 3 are sequential (only one at a time) because all disks are involved in a single I/O.

You can use RAID 0 but once one of your disks fails, then you loose some data and it's difficult to recover.

The RAID can have a spare disk (not used for a time being) that can replace a disk which failed automatically, which means that we don't have to wait for a technician to replace the failed disk manually. It can speed up the recovery time. The only drawback of RAID is its price, RAID 5 for a dozen of disks costs about $6000 which is still a high price.

Many big companies have the same philosophy as RAID. In RAID we use many commodity disks to provide fast and reliable storage, which is still keenly priced when compared to one huge disk (even taking into account the RAID price). The huge IT companies use many commodity machines instead of beefy servers.

Mentoring

"The future of science…so important to the health and prosperity of the world, depends on the skillful mentoring of each new generation by the one that precedes it." Bruce Alberts