Class RobotsTxtBuilder

Namespace

SunamoRobotsTxt

Assembly

SunamoRobotsTxt.dll

Builds and parses robots.txt files.

public class RobotsTxtBuilder

Inheritance

object

RobotsTxtBuilder

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Extension Methods

FuncExtensions.AsFuncFast<T>(T)

FuncExtensions.AsFunc<T>(T)

ObjectExtensions.GetStackTrace(object)

ReClasser.FixMeUp(object)

Constructors

RobotsTxtBuilder(IEnumerable<string>)

Parses the given lines of a robots.txt file.

public RobotsTxtBuilder(IEnumerable<string> enumerable)

Parameters

enumerable IEnumerable<string>

Lines from a robots.txt file to parse.

Properties

Allows

Gets or sets the allowed paths per user agent.

public Dictionary<string, List<string>> Allows { get; set; }

Property Value

Dictionary<string, List<string>>

Disallows

Gets or sets the disallowed paths per user agent.

public Dictionary<string, List<string>> Disallows { get; set; }

Property Value

Dictionary<string, List<string>>

NotRecognizedLines

Gets or sets lines that could not be parsed.

public List<string> NotRecognizedLines { get; set; }

Property Value

List<string>

Sitemaps

Gets or sets the sitemap URLs.

public List<string> Sitemaps { get; set; }

Property Value

List<string>

Methods

AddOrCreateIfDontExists(IDictionary<string, List<string>>, string, string)

Adds a value to the list for the given key, creating the list if it does not exist.

public static void AddOrCreateIfDontExists(IDictionary<string, List<string>> dictionary, string key, string value)

Parameters

dictionary IDictionary<string, List<string>>

The dictionary to add to.

key string

The key to add the value under.

value string

The value to add.

AddOrCreateIfDontExists<TKey, TValue>(IDictionary<TKey, List<TValue>>, TKey, TValue)

Adds a value to the list for the given key, creating the list if it does not exist.

public static void AddOrCreateIfDontExists<TKey, TValue>(IDictionary<TKey, List<TValue>> dictionary, TKey key, TValue value)

Parameters

dictionary IDictionary<TKey, List<TValue>>

The dictionary to add to.

key TKey

The key to add the value under.

value TValue

The value to add.

Type Parameters

TKey

The type of the dictionary key.

TValue

The type of the list elements.

Allow(string, string)

Adds an allow rule for a user agent.

public void Allow(string userAgent, string path)

Parameters

userAgent string

The user agent identifier.

path string

The allowed path.

Disallow(string, string)

Adds a disallow rule for a user agent.

public void Disallow(string userAgent, string path)

Parameters

userAgent string

The user agent identifier.

path string

The disallowed path.

GetValuesOrEmpty<TKey, TValue>(IDictionary<TKey, List<TValue>>, TKey)

Returns the list of values for the given key, or an empty list if the key does not exist.

public static List<TValue> GetValuesOrEmpty<TKey, TValue>(IDictionary<TKey, List<TValue>> dictionary, TKey key)

Parameters

dictionary IDictionary<TKey, List<TValue>>

The dictionary to look up.

key TKey

The key to look up.

Returns

List<TValue>

The list of values, or an empty list.

Type Parameters

TKey

The type of the dictionary key.

TValue

The type of the list elements.

Save(string)

Saves the robots.txt content to a file.

public void Save(string path)

Parameters

path string

The file path to save to.

Sitemap(string)

Adds a sitemap URL.

public void Sitemap(string path)

Parameters

path string

URL of the sitemap.