Core Data has selects

After my last post, there was an interesting side discussion on Twitter (with @macfanatic and @kastiglione) about some alternatives. It was interesting enough that I thought it would make good material for another post. So here ya go!

So, to revisit the last post Core Data in Motion - Chapter 2, we discussed the notion that Core Data doesn't really have the notion of a "select" clause.

What is the impact of this limitation?

If you have a large monolithic model, but just wanted a subset of the attributes for your current view, like, say a MAP, you'd probably want to do something similar to the ActiveRecord query:, :well_name, :surface_latitude, :surface_longitude)

But you can't do that in Core Data. When you execute a Core Data fetch, you get the option of getting back object ids only or whole objects only.

Or can you?

That was the point under discussion on Twitter. There is actually another option, but it comes with its own limitations. So let's take a look at that other option, and see what it gives us, and what it takes away.

If you look at the SDK docs for NSFetchRequest, you will see a method called setPropertiesToFetch. The SDK description for this method is:

Specifies which properties should be returned by the fetch. The 
property descriptions may represent attributes, one-to-one
relationships, or expressions. The name of an attribute or
relationship description must match the name of a description 
on the fetch request’s entity.

    values (Array) — An array of NSPropertyDescription objects that 
    specify which properties should be returned by the fetch.


Hey! So you can get your NSFetchRequest to return only this specified subset of properties! Isn't that just like a select?

Well, not quite. You see, when you run the NSFetchRequest with the propertiesToFetch specified, it will still ignore your propertiesToFetch, unless you have also specified a resultType of NSDictionaryResultType. This is detailed in the answer to this Stack Overflow question.

Then, when you run the NSFetchRequest with some propertiesToFetch and a resultType of NSDictionaryResultType you get back an array of NSDictionary items, NOT your NSManagedObjects. So your results won't have any of the methods you've declared on those NSManagedObjects, like, for instance, if you implemented the MKAnnotation interface so that you can drop those results on an MKMapView (which was my particular issue with this).

I will further note that the Stack Overflow post also indicates that the NSFetchedResultsController doesn't play nicely with NSDictionaryResultsType, which would limit it's usefulness with most standard UITableViewControllers implementations, too.

So... I can't use these results on my map, and it won't play nicely with my list view, either. I'm sure it's useful in some use cases, but its limitations made it useless for both of mine, so this is where NSFetchRequest.propertiesToFetch and I parted company.

Chapter 3 of Core Data in Motion (Preloading Data) is now available, and I've added the information from this post as an update to Chapter 2 as an extra bonus. As always, I welcome your feedback, especially when it helps me improve the book like this discussion has done.

Core Data in Motion - Chapter 2

So, Core Data and relationships. Lots of iOS apps are small, so why wouldn't you just smoosh all your data into one big model?

Performance. Memory. Speed.

With Core Data, you cannot simply retrieve only certain attributes from a object. For instance, in my WIMBY application, we have a rather huge monolithic well table, with about 20 attributes for each well. When we want to display these on a map, we only need 4 of those 20 attributes. A unique identifier, "label" for the pin, and the latitude and longitude values for map coordinates. To be memory/bandwidth efficient, we would only select the 4 attributes we actually need when retrieving the data from the database. In Rails ActiveRecord parlance, that would be something like:, :well_name, :surface_latitude, :surface_longitude)

In this statement, we have only 4 attributes out of our original 20. This saves us approximately 80% of the memory/bandwidth of a full record retrieval, which most programmers would think was a Good Thing™. The only problem here is, we can't do that in Core Data.


Yup, in Core Data, fetches will return either object ID's only or whole objects only. There is no way to refine that fetch with a select clause, like we have in SQL or ORM tools like ActiveRecord.

In our particular case of the well table, Core Data will either fetch entire objects, chewing up 5x the amount of memory that we actually need, or fetch only object ID's, and then lazily fetch each object as it is accessed. In case you haven't run into THIS particular problem before, it's called the N+1 Query problem, and it will have a significant effect on the speed of your application. Your choice here is between MOAR MEMORY or MOAR SLOW.

So how do we make make our object fetches more performant in Core Data?

This is accomplished through decomposing our monolithic models into several distinct models with one-to-one relationships between each model (entity). Each model contains only the information necessary for the view in which it will be used. In the hot-off-the-presses Chapter 2 of Core Data in Motion, we examine, in detail, how to accomplish this.

Core Data with NSFetchedResultsController in RubyMotion

Today we'll be finishing off our series on Core Data in RubyMotion, discussing table view optimization of large amounts of data in your RubyMotion application. If you've missed the earlier posts, you can find them here:

Introduction to Core Data in Motion

Core Data Basics in RubyMotion

Core Data Relationships in RubyMotion

Core Data Pre-loading in RubyMotion

Core Data Load Optimization in RubyMotion

Once again, we turn to Ray Wenderlich for inspiration and instruction. His Core Data tutorial wraps up with a post on the usage of NSFetchedResultsController, so that is where we will end as well.

Why do we want to use NSFetchedResultsController, anyway? What's so special about it? When we started this series, with a relatively small sample dataset, it didn't really need much optimization. Now that we've loaded our database up with all 244,292 wells, it definitely needs some help, because I don't want my customers to wait minutes for the table view to load, which is what it does at this point.

So, we'll need to reduce memory overhead, and improve the response time of our table view, now that we have all that data. Ideally, in a table view, we would only load up the data that is actually visible to the user. And that is exactly what the utility class NSFetchedResultsController provides for us. So let's see how that is accomplished in RubyMotion.

First of all, we create an NSFetchedResultsController. Because this object requires access to the NSManagedObjectContext, which is in our store class, that's where we will put it.

  def fetched_results_controller
    fetch_request = NSFetchRequest.alloc.init
    fetch_request.entity = NSEntityDescription.entityForName('FailedBankInfo', inManagedObjectContext:@context)
    sort = NSSortDescriptor.alloc.initWithKey("details.close_date", ascending: false)
    fetch_request.sortDescriptors = [sort]
    fetch_request.fetchBatchSize = 20


The key to the construction of the NSFetchedResultsController is providing a base NSFetchRequest. This request needs to know which entity is being fetched, and also requires an NSSortDescriptor so it knows in what order to return the requested objects. The fetchBatchSize simply limits the number of objects returned on any single query to the database.

Now that we can create our NSFetchedResultsController, where do we put it? In this case, we will be creating it in our table view controller's viewDidLoad method.

  def viewDidLoad
    error_ptr =
    @fetch_controller = FailedBankStore.shared.fetched_results_controller
    @fetch_controller.delegate = self
    unless @fetch_controller.performFetch(error_ptr)
      raise "Error when fetching banks: #{error_ptr[2].description}"

Here we create the NSFetchedResultsController, set it's delegate to be self, and trigger the initial fetch to populate the table view.

Next, we need to update the table view to get it's data from the NSFetchedResultsController.

  def tableView(tableView, numberOfRowsInSection:section)

  def configureCell(cell, atIndexPath:index)
    bank = @fetch_controller.objectAtIndexPath(index)
    cell.textLabel.text =
    cell.detailTextLabel.text = "#{}, #{bank.state}"
    return cell

  CellID = 'CellIdentifier'
  def tableView(tableView, cellForRowAtIndexPath:indexPath)
    cell = tableView.dequeueReusableCellWithIdentifier(CellID) || UITableViewCell.alloc.initWithStyle(UITableViewCellStyleSubtitle, reuseIdentifier:CellID)
    configureCell(cell, atIndexPath:indexPath)

These methods translate over from Ray's tutorial pretty much intact, without much change, other than the "rubyization".

I did sort of skip a step back there, so let's not forget about that. In viewDidLoad we set the NSFetchedResultsController's delegate to be self. Now, we have to implement the NSFetchedResultsControllerDelegate's signature methods. Ray simply copied his implementation from an Apple sample. I've simply converted his code into a Ruby module.

module NSFetchedResultsControllerDelegate

  def controllerWillChangeContent(controller)

  def controller(controller, didChangeObject:object, atIndexPath:path, forChangeType:type, newIndexPath:new_path)
    tableView = self.tableView
    case type
      when NSFetchedResultsChangeInsert
        tableView.insertRowsAtIndexPaths([new_path], withRowAnimation:UITableViewRowAnimationFade)
      when NSFetchedResultsChangeDelete
        tableView.deleteRowsAtIndexPaths([path], withRowAnimation:UITableViewRowAnimationFade)
      when NSFetchedResultsChangeUpdate
        configureCell(tableView.cellForRowAtIndexPath(path), atIndexPath:path)
      when NSFetchedResultsChangeMove
        tableView.deleteRowsAtIndexPaths([path], withRowAnimation:UITableViewRowAnimationFade)
        tableView.insertRowsAtIndexPaths([new_path], withRowAnimation:UITableViewRowAnimationFade)

  def controller(controller, sectionIndexTitleForSectionName:sectionName)

  def controller(controller, didChangeSection:section, atIndex:index, forChangeType:type)
    case type
      when NSFetchedResultsChangeInsert
        self.tableView.insertSections( NSIndexSet.indexSetWithIndex(index), withRowAnimation:UITableViewRowAnimationFade)
      when NSFetchedResultsChangeDelete
        self.tableView.deleteSections( NSIndexSet.indexSetWithIndex(index), withRowAnimation:UITableViewRowAnimationFade)

  def controllerDidChangeContent(controller)

And then we include the module in our table view controller:

class FailedBankTableViewController < UITableViewController

  include NSFetchedResultsControllerDelegate

It looks like a lot of code, but since it's doubtful you will need to change it much, you should just be able to reuse this module when required.

And that, as they say, is that. We now have a working implementation of NSFetchedResultsController, and the data will only be loaded 20 objects at a time. This speeds things up immensely, and reduces memory usage in our app from get-killed-immediately to just fine ;-) The complete example can be downloaded, compiled, loaded with data and run. Alas, I am unable to provide the "large data load" that I used, as that data is not mine to give away. I encourage you to come up with your own large data set, and plug it in, and see how it works.

If you found this series interesting, you might want to take a look at my Core Data in Motion ebook. It is now available in pre-sales, with a sample chapter.

Core Data Pre-Loading in RubyMotion

<p>This week, we are delving into loading up that data store with a pre-populated set of data.  Once again, Ray Wenderlich's Core Data tutorial offers us a great starting point. <a href="">Core Data on iOS 5 Tutorial: How To Preload and Import Existing Data</a></p>

Read More…

And the survey says...

Thanks to everyone who participated in my Ruby Workshop survey. Although I didn't get a tonne of responses, the ones I did get formed some distinct patterns, so I got what I needed.

In general, you prefer weekends or don't care, so weekend it is.

You distinctly prefer November, with a secondary preference of January, so we'll schedule it that way.

No preferences emerged for location, so I'll just see what's available/comfortable/cheap. ;-) (One write in for Dubai/Muscat… sorry, but probably not)

You really don't like "Mostly lecture", and lean strongly toward a mix of lecture and interactive work-along (which is what we've always done, good to know).

And last but not least, you want the straight up Ruby on Rails for Real workshop most of all, followed by the Rails for iOS Developers, so that will be what we schedule for November and January. A reasonable number of people are wanting the Intro to Ruby as well, and I am certain that Ladies Learning Code will be running that workshop again soon as well, so keep an eye out for that. Oh, and I had one "write in" for RubyMotion. That's on my radar for next year.

Thanks again. I'll post a schedule as soon as I've firmed up the dates. If you are not already on my mailing list, please do signup, and you'll be the first to hear as well as get special, list only, discount codes for workshops.

A Schedule that Fits... ME

Random Thoughts on Developing a Schedule that Fits

It's been a while since I have been "under employed". This time, it is deliberate. When I ended my last contract, I chose to not pursue new opportunities right away, because

  1. It's the middle of summer!
  2. I have numerous personal/business projects that have not had enough care and feeding

While on this kick, working exclusively for myself, I have gotten back to a schedule that fits my life. I intend to keep it that way. These are the keys to my new and improved schedule, that fits ME.

I'm not a morning person, and never will be. Don't get me wrong, I always knew this. But I keep reading all these so-called life-hacks, where they tell me that I need to ignore my email, and the "little stuff", and get to work on my important projects FIRST THING in the morning. Hogwash. My brain doesn't fire on all cylinders first thing in the morning. I use my mornings to catch up on the Inbox (not deal with it all, just get rid of the fluff, and flag the stuff that needs actual work for later). And I catch up on Twitter/Facebook. And I drink my coffee. I eat my breakfast (thanks to Lift app I'm on a 21 day streak there). I open my snail-mail. I do most of this from my iPad, and the location varies. If it's nice out, I can sit out on the patio. If not, it's the comfy couch.

Once I'm done all that, I go to my desk and my computer. Clutter takes away from my focus, so I deal with the bills and things (like the opened mail). Flip quickly thru my flagged email to see if anything urgent needs to get done today. Open up Things (task management app) and review my task list for Today. I'm getting really good at recognizing "Whoa, too many things", and brutally cutting/de-prioritizing. Review my 30x500 weekly action sheet, and make sure I am making the progress I want to make (adding stuff to Things as needed) on my personal projects.

Depending on what time it is (still morning, maybe not), I'll start tackling annoying stuff on my Things list, stuff that I know needs doing today, and will only take 10-30 minutes to finish. I think of it as "clearing the decks". Because, when lunch rolls around, I want to be in a position to be finished what I've started.

I've also started working out again. It's been almost 8 months since I worked out regularly. Just before lunch has been a good time for me to hang that habit. Also, if I don't do it then, I have another window of opportunity at the end of my work day, before supper. If I was just doing it before supper, I know I'd get busy, "in the groove", and fail to stop in time to work out. Having 2 windows makes it that much easier to get it done. I'm also tracking that habit in Lift app. It's been a 10 days of regular workouts, with no "oops, I missed". And it feels great. (And I've almost caught up on my Ruby Tapas videos!)

Ok, now it's afternoon, I've cleared the decks, worked out, eaten lunch. I am FULL of energy. Yes, the afternoon is MY prime time. That's when I can tackle the meaty hard tasks that require intense thinking. And the important stuff that I know is just going to take hours. Because I have hours and hours of uninterrupted time. And I get shit done. Lots of it. Because I have the focus and the time. If I am on a roll, I can just keep working as long as I want. Generally speaking, that isn't all night. Or even all evening. Eventually I'll get tired, or hungry, and start losing focus. That's when it's time to quit. Sometimes, my dog tells me it's time to quit. He's surprisingly good at that.

If I have encountered problems or blockers in my afternoon of work, I'll spend time in the evening surfing and researching on my iPad, looking for answers. I can do that while tired, sitting on the comfy couch. Trying to continue working/focusing while tired has generally been a recipe for frustration, so I resist doing it.

So there you have it. A schedule that fits ME. I think that in order to develop a schedule that fits YOU, you need some time free of outside influence. Those pressures that tell you when you "should" be doing your work, and when you "should" be reading your email. And, of course, you need to have a pile of work that needs to get done, so you don't just fritter the time away.

It's been an interesting 3 weeks. One more to go, then I'll probably need to get back to a new contract (or two or three). This time, though, now that I have a schedule that really fits, I will be resistant to the pressures to conform to other's schedules, and stick to the one where I am most productive and happy.

Ruby/Rails Workshops...

I've been struggling with trying to schedule my next workshop. I know there are lots of people interested, but we had to cancel the last one, because of insufficient signups, which was frustrating both for me, and for the people I had to let down by cancelling.

I'd really rather not do that again so I'm running a (really brief) survey, to see if we can figure out what would be the optimal time for scheduling the next workshop.

If you are in (or near) Calgary, and have thought about learning Ruby and/or Rails, please check out this survey and help me out?

RoR4Real Ruby/Rails Workshop Survey

Core Data Relationships in RubyMotion

So, we've been talking about using Core Data in RubyMotion. If you missed my earlier posts, you can find them here:

Introduction to Core Data in Motion

Core Data Basics in RubyMotion

Today, let's dig into a more meaty problem. Relationships between entities in Core Data. If you google for that, you will undoubtedly end up at Ray Wenderlich's site, and his Core Data Tutorial fo iOS: Getting Started. Which, BTW, is an awesome tutorial that describes exactly how to set up relationships between entities in Core Data… in the Visual Model editor in Xcode.

Right, but we aren't using Xcode (you can, but it's probably not worth the extra effort of duplicating all your data models in Obj-C). We are using RubyMotion, and we want to define our models in code. Thankfully, it's not as difficult as it might seem at first blush. Since I was inspired (and informed), by Ray's tutorial, I have created a RubyMotion project to illustrate this based on his "Failed Banks" example. You can find my project here.

In the basics post we already created Core Data managed objects, defined their attributes, and dealt with the persistent store coordinator, managed object context, and the managed object model. So let's just assume you have all that, and get right down to… two models, how to connect them together.

I was interested to discover that a relationship is simply another property, much like an entity attribute. So, when the entities properties are assigned, you just add the relationships to that array along with the attributes. Seems simple enough.

It's a trap

Unfortunately, it's not that simple after all. In order for two Core Data entities to be connected, each needs to have a relationship defined to the other. The problem here being the old chicken-and-egg. When you define the first entity, the second does not exist, and so the NSRelationshipDescription cannot be fully specified. Argh.

But here's the trick. You CAN create an entity, and add it to the managed object model, WITHOUT specifying any properties (attributes OR relationships) on it. Then we have a nice handy store of all the entities, pre-defined, and can proceed to lazily specify the relationships. Let's see what that looks like.

First, I have re-factored my models, so that the entity creation does not add attributes at the same time.

  def self.entity
    @entity ||= begin
      # Create the entity for our managed object class
      entity = NSEntityDescription.alloc.init = name
      entity.managedObjectClassName = name

As you can see here, this code is looking pretty non-specific to the model, and can easily be factored out. In fact, that's where most of the RubyMotion specific modelling gems that support Core Data start. I'll leave that as an exercise for you.

Next we have separate definitions for the attributes:

    @attributes ||= [
      {:name => 'name', :type => NSStringAttributeType, :default => nil, :optional => false},
      {:name => 'city', :type => NSStringAttributeType, :default => nil, :optional => false},
      {:name => 'state', :type => NSStringAttributeType, :default => nil, :optional => false},

and the relationships:

  def self.relationships
    @relationships ||= [
        :name => 'details', 
        :destination => 'FailedBankDetails', 
        :inverse => 'info', 
        :optional => true, :transient => false, :indexed => false, :ordered => true, 
        :min => 1, :max => 1, :del => NSCascadeDeleteRule

We have similar atrributes and relationship defined on the FailedBankDetails entity, which is the inverse of this relationship.

Ok, now that we have entities, attributes, and relationships specified, let's jump over to our store object, where we create the NSManagedObjectModel:

    model ||= NSManagedObjectModel.alloc.init.tap do |m|
      m.entities = [FailedBankInfo, FailedBankDetails].collect {|c| c.entity}
      m.entities.each {|entity| set_entity_properties(entity,m)}

Here we have added all the entities in our data model (all two of them) to the mom. Once those entities exist, they can be used to complete the creation of the relationships.

  def set_entity_properties(entity, model)
    # set up attributes
    managed_object_class = Object.const_get(entity.managedObjectClassName)
    entities = model.entitiesByName

    attributes = managed_object_class.attributes.collect do |attr|
      property = NSAttributeDescription.alloc.init = attr[:name]
      property.attributeType = attr[:type]
      property.defaultValue = attr[:default]
      property.optional = attr[:optional]
    # set up relationships
    relationships = do |rel|
      relation = NSRelationshipDescription.alloc.init = rel[:name]
      relation.destinationEntity = entities[rel[:destination]]
      relation.inverseRelationship = entities[rel[:inverse]]
      relation.optional = rel[:optional] || false
      relation.transient = rel[:transient] || false
      relation.indexed = rel[:indexed] || false
      relation.ordered = rel[:ordered] || false
      relation.minCount = rel[:min] || 1
      relation.maxCount = rel[:max] || 1 # NSIntegerMax
      relation.deleteRule = rel[:del] || NSNullifyDeleteRule # NSNoActionDeleteRule NSNullifyDeleteRule NSCascadeDeleteRule
    # assign properties = attributes + relationships

And voila! We have a fully specified data model, with relationships and inverse relationships.

The complete example can be downloaded and run. It's not actually that exciting without any data loaded. I'm sure you won't be too surprised to discover that is the next topic that we will be covering.

Until then…

You might want to sign up for my Core Data in Motion ebook. Everyone who signs up will get advance notice, coupon deals, and a free chapter from the book when it is complete.

Rails for iOS Developers

Just in case anyone missed it, I will be teaching and speaking at NSScotland in October.  

On tutorial day, my partner in crime, Tim Breitkreutz, and I will be teaching the popular Rails for iOS Developers workshop.  If you are an iOS developer, and you need a website and back end for your iOS apps, this workshop is for you!  It sold out at Ruby Conf AU in February, so sign up soon!

At the conference in general, I will be speaking about using Core Data in RubyMotion, an introduction to Core Data, in code, at it's most basic levels, and beyond.  Even if you don't use RubyMotion, you will undoubtedly learn something new about Core Data, from it's pure, no magical Xcode tools approach.

I'm looking forward to Scotland, and I hope to see a lot of you there!