Sunday, 28 April 2013

How to read a CSV file in octave and set empty cells to defaults

if foo.csv is a csv file with some empty values and a header row (also with some empty values) you can fill in defaults within octave using this method:

d = csv2cell('foo.csv')
e = cellfun('isempty', d)

eh = false(size(e))
en = false(size(e))

eh(1, :) = e(1, :)
en(2:end, :) = e(2:end, :)
d(eh) = {'No header name'}
d(en) = {NaN}

Sunday, 26 August 2012

List makefile targets

This is a program that lists the targets in a makefile (using GNU make).

Hopefully it's fairly clear. There is some complexity for performance (all filtering is done by sed).

A version of this script is now included in bash completion from debian although it might not be released as of this writing. That version doesn't output the full target name, it stops at directory separators to make the tab completion feature more comfortable but this version simply outputs all targets in full, once each.

It doesn't output implicit targets nor any targets listed in the .INTERMEDIATE rule. It *does* output generated targets in complex makefiles.

If your makefile includes others that are non-existent targets themselves then they will be generated before this script finishes running. You can avoid that if your makefile is written in an appropriate manner.


  /^# Make data base/,/^# Files/d             # skip until files section
  /^# Not a target/,+1          d             # following target isnt
  /^\.PHONY:/                   d             # special target
  /^\.SUFFIXES:/                d             # special target
  /^\.DEFAULT:/                 d             # special target
  /^\.PRECIOUS:/                d             # special target
  /^\.INTERMEDIATE:/            d             # special target
  /^\.SECONDARY:/               d             # special target
  /^\.SECONDEXPANSION:/         d             # special target
  /^\.DELETE_ON_ERROR:/         d             # special target
  /^\.IGNORE:/                  d             # special target
  /^\.LOW_RESOLUTION_TIME:/     d             # special target
  /^\.SILENT:/                  d             # special target
  /^\.EXPORT_ALL_VARIABLES:/    d             # special target
  /^\.NOTPARALLEL:/             d             # special target
  /^\.ONESHELL:/                d             # special target
  /^\.POSIX:/                   d             # special target
  /^\.NOEXPORT:/                d             # special target
  /^\.MAKE:/                    d             # special target

# The stuff above here describes lines that are not
#  explicit targets or not targets other than special ones
# The stuff below here decides whether an explicit target
#  should be output.

  /^[^#\t:=%]+:([^=]|$)/ {                    # found target block
    h                                         # hold target
    d                                         # delete line
  /^# File is an intermediate prerequisite/ { # nope
    s/^.*$//;x                                # unhold target
    d                                         # delete line
  /^([^#]|$)/ {                               # end of target block
    s/^.*$//;x                                # unhold target
    s/:.*$//p                                 # write current target
    d                                         # hide any bugs

make -npq .DEFAULT 2>/dev/null | sed -n -r "$SCRIPT" \
  | sort | uniq

This Makefile demonstrates the relative completeness of the above script

As you add .cc files to a subfolder called "test", eg "test/" targets are dynamically defined for "test/a.success", "test/a.result", "test/a.bin".

A target "test/a.o" is also defined but that is incidental for automatic dependencies and for setting custom variables for files in the test folder. test/a.o is dynamically listed as an intermediate target so the above script does not list it.

.PHONY: all test/%.success
.PRECIOUS: test/%.bin test/%.d

TOP := $(shell pwd)
CXXFLAGS = -std=gnu++0x -ggdb -O0

TESTS:=$(patsubst test/,%,$(wildcard test/*.cc))
TESTSUCCESSES:=$(patsubst %,test/%.success,$(TESTS))


$(foreach SUCCESS,$(TESTSUCCESSES),$(eval $(SUCCESS):))
$(foreach RESULT,$(patsubst %.success,%.result,$(TESTSUCCESSES)),$(eval $(RESULT):))
$(foreach BIN,$(patsubst %.success,%.bin,$(TESTSUCCESSES)),$(eval $(BIN):))

.INTERMEDIATE: $(patsubst %.success,%.o,$(TESTSUCCESSES))

test/%.o: CXXFLAGS += -I test -I.

test/%.o: test/
 g++ -c $(CXXFLAGS) -o $@ $<
 @g++ -MM $(CXXFLAGS) $< | sed -e '1s/:/: \\\n/' | sed -e '1s:^:test/:' > test/$*.d

test/%.bin: test/%.o
 g++ $(CXXFLAGS) -o $@ $^

test/%.result: test/%.bin
 $< > $@

test/%.success: test/%.result
 @test `grep -c ^fail $<` -eq 0 || { cat $< 1>&2; false; }
 rm -f *.o test/*.o test/*.s test/*.bin test/*.result test.result test/*.d *.d test/*.pretest

include $(patsubst %.o,%.d,$(wildcard test/*.o))

Saturday, 28 April 2012

Did you know that If any application on your android device has had access to your data at google that there is no server side restriction on what other apps on that device can access? Strange but true... And frighteningly bad security by Google.

Saturday, 14 April 2012

Diet Graph

This is a graph of my weight loss efforts. The orange line is an estimate of my excess weight based on occasional samples (green), a model of my BMR (not shown), and my intake and exercise figures (blue and red respectively).

Saturday, 16 July 2011

windows installer and wix article index - blog entries by robman

The considerations of an installer - the domains of study

Basics about a "Component" in Windows Installer terms
rules of thumb can be derived from this

Historical commentary about the creation of Orca

More detail about what a component is and how to use/defined them correctly
rules of thumb can be derived from this

What the MSI file is and what it can describe
  • "the Darwin team decided to build a custom relational database"
  • "MSI files are actually little databases laid out in a structured storage file"
  • "The MSI file uses separate streams for each of the tables in the database."
  • "For example, streams are used to store things like UI graphics, CustomAction DLLs, and even the binaries to be installed in many cases."
  • "Also, sub-storages are used to nest one MSI file inside another MSI file (note: you should never do this, but I'll talk about nested installs another day)."
  • "the names of streams can only be something around 63 characters." - "can cause some really wacky error messages"
  • "add then delete data to a structured storage file, the file maintains its largest size" - "so it is possible to end up with bloated MSI files if you're not careful"
  • "structured storage files don't handle multiple writers well at all. For example, open an MSI file in Orca then try to install the MSI by double clicking on it. You'll get a lovely message box"
  • "there is a single stream in the MSI file that holds all the strings. This stream is called the string pool contains a single entry for each unique string. That way a string column in a table is just an integer offset into the string pool."
  • "MSI file format is not Unicode" - "there is a pretty detailed topic in MSDN about localizing MSI files"
  • "you have to deal with codepages when storing localized strings in a MSI file"
  • "Next blog I'll actually try to answer Jim's question about creating custom tables in a MSI file"
  • Q: did he do that?

More what the MSI file is and what it can describe
  • includes link to interesting application compatibility problems list
  • references msiinfo.exe, dfview.exe

Some thoughts on packaging of deployments and their parameters
Really looking at convincing the datacentre to move to msi

"You can't put managed code in a custom action" - the reason why, and also why you shouldn't expect to any time soon
  • Note that this is wrong due to DTF release, but has an interesting discussion of the issues
  • Will make you realise why you should use DTF for this rather than your own solution

How to defined an uninstall shortcut - you probably shouldn't even if the prodman asks - but its a neat example of wix
Includes discussion of weird facts which might have wider utility

A little computer science regarding wix's programming paradigm

If you need the same file in multiple places - this keeps the package small
  • Two indexes to one file in the cab
  • Alternative to DuplicateFiles action (because it allows integrity checking like separate installed files)
  • Regular duplication functionality will not copy files out of the Global Assembly Cache so the .NET Framework scenario still requires smart cabbing

How to make custom actions the right way
Reading List:
  • MSI SDK Topic about Custom Actions
  • blog entry why Script Custom Actions suck
  • another reason why Script Custom Actions suck

Why your setup should be done early and included in testing by module
  • before even writing tests that one should write the setup logic
  • encourage the development of code that can be better

more on doing custom actions right
  • When I'm presented with the need to write a custom action I often mentally classify the work into one of three or so different buckets:
    1. Internal-only custom actions
      Transform properties or tables into temporary properties or tables
    2. Read-only custom actions
      Gather system data into temporary properties or tables
    3. System modification custom actions
      the most complex custom action to write
      • usually have to be deferred (which is a fascinating set of future topics)
      • Q: Does he cover "deferred" actions anywhere?
  • Eventually, [He'll] even get around to writing code or dissecting existing custom action code.
  • Q: Does he?

why you should really try to avoid custom actions and design to simplify your installer
  • "Having written a great number of custom actions for the WiX toolset (to try and help address c. above) I can assure you that getting the code right in all cases (especially patching) is extremely difficult. If you look at the last bugs in WiX v2, you'll find they have all been custom action issues."
  • "So, ultimately, I would encourage all setup developers to be very skeptical about custom actions and continually work to reduce the amount of custom code in your installation package. There are few setup experiences more stable than an application that simply needs a bunch of files installed. Get in the habit of questioning the value of every setup requirement beyond copying files and you may be surprised how stable things can be. "

surprises and interesting things with windows installer 4.5
  • The new msidbCustomActionTypePatchUninstall value is a bit of a pain
  • Cannot use 4.5 patching feature for msi's for pre 4.5
  • ICE45 error. Fortunately, we can suppress ICE45 in WiX v3 by added "-sice:ICE45" to the light.exe command-line.
  • This is a big problem for authoring tools and developers so I expect the Windows Installer team is going to have to come up with a different solution (like add a new column on the end of the CustomAction table called TypeEx)
  • Q: Did they?
  • Other msi 4.5 features mentioned
  • Q: What's the status of msi 4.5

more on getting custom actions right
  • your custom action should read data out of a table and act based on those declarative instructions
  • He says he'll walk step-by-step through his wordy example
  • Q: Has he? Where?

ASP.NET MVC related stuff - NOT installer

tools for editing/creating wix files
WiX v3 editors:
  • Setup Factory for Windows Installer (commercial) - was the first commercial editor built on the WiX v3 toolset.
  • Votive (open source) - Visual Studio/MSBuild plugin. Being included in v3.5.
  • WixEdit (open source) - continues to adapt to the changes in the WiX toolset.

Why you can now do custom actions with managed code
  • DTF (Deployment Tools Foundation) now in Wix
  • Supports managed code custom actions
  • Q: Does DTF support analysis/comparison of msi and data streams?

For some reason this is important enough to get a blog entry with a link. dunno why

Some information on how to troubleshoot msi installers

Introduction to the Burn bootstrapper

Glossary for the Wix "Burn" bootstrapper
  • Burn would work as a single self extracting executable.
  • "Package": Unit of change
  • "Payload": A file of data for a package
  • "Container": A way to combine payloads in a single file
  • "Attached container": Container embedded at the end of the bootstrapper file itself
  • "Detached container": Container in a file outside the bootstrapper
  • Q: Are there other types of container?

A bit about the future - Wix 4
  • some detail on features for wix v4.0
  • Language improvements:
    • MajorUpgrade element
    • Component element simplification
    • Small breaking changes
  • Add migration to WixCop.exe
  • Code generation, ala heat.exe

What does Name="SourceDir" do?

What's in Wix 3.5 - changes from earlier plans
  • Votive (full vs2010 plugin) -> wix v3.5
  • IIS7 custom action -> wix v3.5 or v3.6
  • Burn -> wix v3.6 (planned to be stable end sept 2011)
  • simplification -> wix v4.0

Something about Wix UI and language support
  • dialog localisations are in .wxl files
  • patch dialogs need localising like initial installer dialogs
  • Q: can I/how can I limit standard dialog localisations to only use ones I am happy to create for my own dialogs?
  • List of localisations for the patch dialogs at the time of writing

Suggestions on how to work with the wix source code
  • wix is maintained via the mercurial (hg) version control system
  • instructions on using mercurial to work with the wix repository

Using Burn bootstrapper for corporate customers
  • it seems burn downloads from the internet
  • it can "download" from a local location instead
  • it ought to and will have a -layout switch that will download to make local copies


Case study with wixlibs
  • a .wxs file should only appear in one .wixproj file.
  • common elements can go into .wixlib files to be referenced by the .wxs file.

Sunday, 9 January 2011

Weightloss Techniques Brainstorm

This is a brainstorm of ideas for how to lose weight while maintaining a modern bachelor lifestyle.

Please comment with other suggestions
Entries will have a heading with a basic description of a technique, then notes on why, a category, and ways of making the technique compete with foods that could otherwise interfere.

Avoid fruit juice - fruit is for eating, water is for drinking.

200ml (one small glass) of even squeezed orange juice with bits left in has about a quarter of a normal person's sugar, in one small liquid hit - which, even though it's technically fruit, is no good for the body's metabolic behaviour

Category: Mildness Acceptance

Competitiveness: how long do peeled fruits last after peeling - ie, can they be peeled in a batch and eaten simply on-demand?

Avoid bitter foods and drinks

It is very easy to eat sugary food afterwards without thinking about it to remove the bitter sensation

Category: Craving Suppression

Competitiveness: No Options, Picture going to the coffee machine, saying "no" and putting the mug back - then hopefully you'll remember not to drink it.

Avoid all sugary foods and treats!

Sugar causes huge disruption in perception. It makes other foods taste bland, causes intense needs for more sugar, causes poor moods that make you lose motivation to be mindful of weightloss considerations

Category: Craving Suppression, Mindfulness, Mildness Acceptance

Competitiveness: No Options, Picture picking up one of your usual snacks or imagine the sensation of intense sweetness, saying "no" and putting the food in the bin.

Eat Soup... Not that horrible salty crap from a can or packet...

But at the weekend just fry up a whole heap of lean meat, onions, garlic. Boil loads of root vegetables. Drain the root veg and reduce the water. Blitz all the ingredients. Add the water in once it reaches a low volume, add a bit of skimmed milk to make your preferred texture, lastly add salt and pepper to taste. Store in cuboid portioned food storage boxes and pack into the fridge/freezer. Remember to only have three days worth in the fridge at a time.

Category: Appetite Suppression, Mindfulness

Competitiveness: Always do this like clock work every weekend and it competes with other foods come lunchtime. Do not take it to work in advance, only on the day... otherwise, halfway through the week you'll forget and by some crap instead.

Eat breakfast every day... Lean Protein, low refined carbs, no packet cereal.

Get a hotplate grill (eg, george foreman). Get thinly sliced meat (eg, slice a chicken breast into two large flat steaks and the tender). Cook quickly till just barely cooked - not till dry. Put between bread for practical reasons with a little bit of squeezy mayo (not weird low-fat mayo crap - it's unnecessary

Category: Appetite Suppression

Competitiveness: This needs to compete with not eating breakfast and thus developing cravings. To do this you need to ensure there is preparation and cleanup space available, that the hotplate is always available and kept clean.

Thursday, 23 December 2010

'Twas twilight on Christmas, when all through the house...

'Twas twilight on Christmas, when all through the house
not an adult head sober, "thanks famous <hic> grouse!"

Mum's stockings were laddered beyond all repair,
Her hopes for a jolly day turned to despair.
The children had wrestled each other to shreds,
While driven by sugar from what they were fed.

Gran was annoyed from the afternoon's chatter,
she'd heard not a word of the Queens royal patter.
Then Aunt Mable shouts from her unclosing trap,
"What's that on the telly! The same boring crap!"

My plans for a warm family party were smashed,
So outside I ran from the chavvy white trash,
Where yellow words written in yesterday's snow
were pointing the way to our drunk Uncle Joe.

Then lo and behold but the sight of the year,
Uncle Joe staggers still holding his beer.
My shame overwhelms me, I walk away quick,
He never knows what he should do with his drink.

I'd thought this time everyone wouldn't be lame
but just like the other ones, this year's the same.